Image-based self-serve beverage detection and assignment

ABSTRACT

An image sensor is positioned such that a field-of-view of the sensor encompasses portion of a beverage machine. The field-of-view includes a first zone associated with operating the beverage machine to dispense the beverage and a second zone in which a cup is placed to receive the beverage. A beverage assignment subsystem receives angled-view images from the image sensor. An event associated with an object entering one or both of the first zone and the second zone is detected. In image(s) associated with a start of the event, it is determined that both a hand of a person enters the first zone and the cup is placed in the second zone. If the cup remained in the second zone for at least a threshold time, the beverage is assigned to the person whose hand entered the first zone.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of:

U.S. patent application Ser. No. 16/663,710 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “TOPVIEW OBJECTTRACKING USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,766 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “DETECTING SHELFINTERACTIONS USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,451 filed Oct. 25, 2019, bySarath Vakacharla et al., and entitled “TOPVIEW ITEM TRACKING USING ASENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,794 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “DETECTING AND IDENTIFYINGMISPLACED ITEMS USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/663,822 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM”;

U.S. patent application Ser. No. 16/941,415 filed Jul. 28, 2020, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM USING A MARKER GRID”, which is a continuation of U.S.patent application Ser. No. 16/794,057 filed Feb. 18, 2020, by ShahmeerAli Mirza et al., and entitled “SENSOR MAPPING TO A GLOBAL COORDINATESYSTEM USING A MARKER GRID”, now U.S. Pat. No. 10,769,451 issued Sep. 8,2020, which is a continuation of U.S. patent application Ser. No.16/663,472 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “SENSOR MAPPING TO A GLOBAL COORDINATE SYSTEM USING A MARKERGRID”, now U.S. Pat. No. 10,614,318 issued Apr. 7, 2020;

U.S. patent application Ser. No. 16/663,856 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SHELF POSITION CALIBRATION INAGLOBAL COORDINATE SYSTEM USING A SENSOR ARRAY”;

U.S. patent application Ser. No. 16/664,160 filed Oct. 25, 2019, byTrong Nghia Nguyen et al., and entitled “CONTOUR-BASED DETECTION OFCLOSELY SPACED OBJECTS”;

U.S. patent application Ser. No. 17/071,262 filed Oct. 15, 2020, byShahmeer Ali Mirza et al., and entitled “ACTION DETECTION DURING IMAGETRACKING”, which is a continuation of U.S. patent application Ser. No.16/857,990 filed Apr. 24, 2020, by Shahmeer Ali Mirza et al., andentitled “ACTION DETECTION DURING IMAGE TRACKING”, which is acontinuation of U.S. patent application Ser. No. 16/793,998 filed Feb.18, 2020, by Shahmeer Ali Mirza et al., and entitled “ACTION DETECTIONDURING IMAGE TRACKING”, now U.S. Pat. No. 10,685,237 issued Jun. 16,2020, which is a continuation of U.S. patent application Ser. No.16/663,500 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “ACTION DETECTION DURING IMAGE TRACKING”, now U.S. Pat. No.10,621,444 issued Apr. 14, 2020;

U.S. patent application Ser. No. 16/857,990 filed Apr. 24, 2020, byShahmeer Ali Mirza et al., and entitled “ACTION DETECTION DURING IMAGETRACKING”, which is a continuation of U.S. patent application Ser. No.16/793,998 filed Feb. 18, 2020, by Shahmeer Ali Mirza et al., andentitled “ACTION DETECTION DURING IMAGE TRACKING”, now U.S. Pat. No.10,685,237 issued Jun. 16, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/663,500 filed Oct. 25, 2019, by Shahmeer AliMirza et al., and entitled “ACTION DETECTION DURING IMAGE TRACKING”, nowU.S. Pat. No. 10,621,444 issued Apr. 14, 2020;

U.S. patent application Ser. No. 16/664,219 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “OBJECT RE-IDENTIFICATION DURINGIMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,269 filed Oct. 25, 2019, byMadan Mohan Chinnam et al., and entitled “VECTOR-BASED OBJECTRE-IDENTIFICATION DURING IMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,332 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “IMAGE-BASED ACTION DETECTIONUSING CONTOUR DILATION”;

U.S. patent application Ser. No. 16/664,363 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “DETERMINING CANDIDATE OBJECTIDENTITIES DURING IMAGE TRACKING”;

U.S. patent application Ser. No. 16/664,391 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “OBJECT ASSIGNMENT DURING IMAGETRACKING”;

U.S. patent application Ser. No. 16/664,426 filed Oct. 25, 2019, bySailesh Bharathwaaj Krishnamurthy et al., and entitled “AUTO-EXCLUSIONZONE FOR CONTOUR-BASED OBJECT DETECTION”;

U.S. patent application Ser. No. 16/884,434 filed May 27, 2020, byShahmeer Ali Mirza et al., and entitled “MULTI-CAMERA IMAGE TRACKING ONA GLOBAL PLANE”, which is a continuation of U.S. patent application Ser.No. 16/663,533 filed Oct. 25, 2019, by Shahmeer Ali Mirza et al., andentitled “MULTI-CAMERA IMAGE TRACKING ON A GLOBAL PLANE”, now U.S. Pat.No. 10,789,720 issued Sep. 29, 2020;

U.S. patent application Ser. No. 16/663,901 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “IDENTIFYING NON-UNIFORM WEIGHTOBJECTS USING A SENSOR ARRAY”; and

U.S. patent application Ser. No. 16/663,948 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SENSOR MAPPING TO A GLOBALCOORDINATE SYSTEM USING HOMOGRAPHY”, which are all incorporated hereinby reference.

TECHNICAL FIELD

The present disclosure relates generally to object detection andtracking, and more specifically, to image-based self-serve beveragedetection and assignment.

BACKGROUND

Identifying and tracking objects within a space poses several technicalchallenges. Existing systems use various image processing techniques toidentify objects (e.g. people). For example, these systems may identifydifferent features of a person that can be used to later identify theperson in an image. This process is computationally intensive when theimage includes several people. For example, to identify a person in animage of a busy environment, such as a store, would involve identifyingeveryone in the image and then comparing the features for a personagainst every person in the image. In addition to being computationallyintensive, this process requires a significant amount of time whichmeans that this process is not compatible with real-time applicationssuch as video streams. This problem becomes intractable when trying tosimultaneously identify and track multiple objects. In addition,existing systems lack the ability to determine a physical location foran object that is located within an image.

SUMMARY

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which person in the store removed the itemand to charge that person for the item without needing to ring up theitem at a register. In other words, the person can walk into the store,take items, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. Increasing the number of cameras poses atechnical challenge because each camera only provides a field of viewfor a portion of the physical space. This means that information fromeach camera needs to be processed independently to identify and trackpeople and objects within the field of view of a particular camera. Theinformation from each camera then needs to be combined and processed asa collective in order to track people and objects within the physicalspace.

The system disclosed in the present application provides a technicalsolution to the technical problems discussed above by generating arelationship between the pixels of a camera and physical locationswithin a space. The disclosed system provides several practicalapplications and technical advantages which include 1) a process forgenerating a homography that maps pixels of a sensor (e.g. a camera) tophysical locations in a global plane for a space (e.g. a room); 2) aprocess for determining a physical location for an object within a spaceusing a sensor and a homography that is associated with the sensor; 3) aprocess for handing off tracking information for an object as the objectmoves from the field of view of one sensor to the field of view ofanother sensor; 4) a process for detecting when a sensor or a rack hasmoved within a space using markers; 5) a process for detecting where aperson is interacting with a rack using a virtual curtain; 6) a processfor associating an item with a person using a predefined zone that isassociated with a rack; 7) a process for identifying and associatingitems with a non-uniform weight to a person; and 8) a process foridentifying an item that has been misplaced on a rack based on itsweight.

In one embodiment, the tracking system may be configured to generatehomographies for sensors. A homography is configured to translatebetween pixel locations in an image from a sensor (e.g. a camera) andphysical locations in a physical space. In this configuration, thetracking system determines coefficients for a homography based on thephysical location of markers in a global plane for the space and thepixel locations of the markers in an image from a sensor. Thisconfiguration will be described in more detail using FIGS. 2-7 .

In one embodiment, the tracking system is configured to calibrate ashelf position within the global plane using sensors. In thisconfiguration, the tracking system periodically compares the currentshelf location of a rack to an expected shelf location for the rackusing a sensor. In the event that the current shelf location does notmatch the expected shelf location, then the tracking system uses one ormore other sensors to determine whether the rack has moved or whetherthe first sensor has moved. This configuration will be described in moredetail using FIGS. 8 and 9 .

In one embodiment, the tracking system is configured to hand offtracking information for an object (e.g. a person) as it moves betweenthe field of views of adjacent sensors. In this configuration, thetracking system tracks an object's movement within the field of view ofa first sensor and then hands off tracking information (e.g. an objectidentifier) for the object as it enters the field of view of a secondadjacent sensor. This configuration will be described in more detailusing FIGS. 10 and 11 .

In one embodiment, the tracking system is configured to detect shelfinteractions using a virtual curtain. In this configuration, thetracking system is configured to process an image captured by a sensorto determine where a person is interacting with a shelf of a rack. Thetracking system uses a predetermined zone within the image as a virtualcurtain that is used to determine which region and which shelf of a rackthat a person is interacting with. This configuration will be describedin more detail using FIGS. 12-14 .

In one embodiment, the tracking system is configured to detect when anitem has been picked up from a rack and to determine which person toassign the item to using a predefined zone that is associated with therack. In this configuration, the tracking system detects that an itemhas been picked up using a weight sensor. The tracking system then usesa sensor to identify a person within a predefined zone that isassociated with the rack. Once the item and the person have beenidentified, the tracking system will add the item to a digital cart thatis associated with the identified person. This configuration will bedescribed in more detail using FIGS. 15 and 18 .

In one embodiment, the tracking system is configured to identify anobject that has a non-uniform weight and to assign the item to aperson's digital cart. In this configuration, the tracking system uses asensor to identify markers (e.g. text or symbols) on an item that hasbeen picked up. The tracking system uses the identified markers to thenidentify which item was picked up. The tracking system then uses thesensor to identify a person within a predefined zone that is associatedwith the rack. Once the item and the person have been identified, thetracking system will add the item to a digital cart that is associatedwith the identified person. This configuration will be described in moredetail using FIGS. 16 and 18 .

In one embodiment, the tracking system is configured to detect andidentify items that have been misplaced on a rack. For example, a personmay put back an item in the wrong location on the rack. In thisconfiguration, the tracking system uses a weight sensor to detect thatan item has been put back on a rack and to determine that the item isnot in the correct location based on its weight. The tracking systemthen uses a sensor to identify the person that put the item on the rackand analyzes their digital cart to determine which item they put backbased on the weights of the items in their digital cart. Thisconfiguration will be described in more detail using FIGS. 17 and 18 .In one embodiment, the tracking system is configured to determine pixelregions from images generated by each sensor which should be excludedduring object tracking. These pixel regions, or “auto-exclusion zones,”may be updated regularly (e.g., during times when there are no peoplemoving through a space). The auto-exclusion zones may be used togenerate a map of the physical portions of the space that are excludedduring tracking. This configuration is described in more detail usingFIGS. 19 through 21 .

In one embodiment, the tracking system is configured to distinguishbetween closely spaced people in a space. For instance, when two peopleare standing, or otherwise located, near each other, it may be difficultor impossible for previous systems to distinguish between these people,particularly based on top-view images. In this embodiment, the systemidentifies contours at multiple depths in top-view depth images in orderto individually detect closely spaced objects. This configuration isdescribed in more detail using FIGS. 22 and 23 .

In one embodiment, the tracking system is configured to track peopleboth locally (e.g., by tracking pixel positions in images received fromeach sensor) and globally (e.g., by tracking physical positions on aglobal plane corresponding to the physical coordinates in the space).Person tracking may be more reliable when performed both locally andglobally. For example, if a person is “lost” locally (e.g., if a sensorfails to capture a frame and a person is not detected by the sensor),the person may still be tracked globally based on an image from a nearbysensor, an estimated local position of the person determined using alocal tracking algorithm, and/or an estimated global position determinedusing a global tracking algorithm. This configuration is described inmore detail using FIGS. 24A-C through 26.

In one embodiment, the tracking system is configured to maintain arecord, which is referred to in this disclosure as a “candidate list,”of possible person identities, or identifiers (i.e., the usernames,account numbers, etc. of the people being tracked), during tracking. Acandidate list is generated and updated during tracking to establish thepossible identities of each tracked person. Generally, for each possibleidentity or identifier of a tracked person, the candidate list alsoincludes a probability that the identity, or identifier, is believed tobe correct. The candidate list is updated following interactions (e.g.,collisions) between people and in response to other uncertainty events(e.g., a loss of sensor data, imaging errors, intentional trickery,etc.). This configuration is described in more detail using FIGS. 27 and28 .

In one embodiment, the tracking system is configured to employ aspecially structured approach for object re-identification when theidentity of a tracked person becomes uncertain or unknown (e.g., basedon the candidate lists described above). For example, rather thanrelying heavily on resource-expensive machine learning-based approachesto re-identify people, “lower-cost” descriptors related to observablecharacteristics (e.g., height, color, width, volume, etc.) of people areused first for person re-identification. “Higher-cost” descriptors(e.g., determined using artificial neural network models) are used whenthe lower-cost descriptors cannot provide reliable results. Forinstance, in some cases, a person may first be re-identified based onhis/her height, hair color, and/or shoe color. However, if thesedescriptors are not sufficient for reliably re-identifying the person(e.g., because other people being tracked have similar characteristics),progressively higher-level approaches may be used (e.g., involvingartificial neural networks that are trained to recognize people) whichmay be more effective at person identification but which generallyinvolve the use of more processing resources. These configurations aredescribed in more detail using FIGS. 29 through 32 .

In one embodiment, the tracking system is configured to employ a cascadeof algorithms (e.g., from more simple approaches based on relativelystraightforwardly determined image features to more complex strategiesinvolving artificial neural networks) to assign an item picked up from arack to the correct person. The cascade may be triggered, for example,by (i) the proximity of two or more people to the rack, (ii) a handcrossing into the zone (or a “virtual curtain”) adjacent to the rack,and/or (iii) a weight signal indicating an item was removed from therack. In yet another embodiment, the tracking system is configured toemploy a unique contour-based approach to assign an item to the correctperson. For instance, if two people may be reaching into a rack to pickup an item, a contour may be “dilated” from a head height to a lowerheight in order to determine which person's arm reached into the rack topick up the item. If the results of this computationally efficientcontour-based approach do not satisfy certain confidence criteria, amore computationally expensive approach may be used involving poseestimation. These configurations are described in more detail usingFIGS. 33A-C through 35.

In one embodiment, the tracking system is configured to track an itemafter it exits a rack, identify a position at which the item stopsmoving, and determines which person is nearest to the stopped item. Thenearest person is generally assigned the item. This configuration may beused, for instance, when an item cannot be assigned to the correctperson even using an artificial neural network for pose estimation. Thisconfiguration is described in more detail using FIGS. 36A, 36B, and 37 .

In one embodiment, the tracking system is configured to detect when aperson removes or replaces an item from a rack using triggering eventsand a wrist-based region-of-interest (ROI). In this configuration, thetracking system uses a combination of triggering events and ROIs todetect when an item has been removed or replaced from a rack, identifiesthe item, and then modifies a digital cart of a person that is adjacentto the rack based on the identified item. This configuration isdescribed in more detail using FIGS. 39-42 .

In one embodiment, the tracking system is configured to employ a machinelearning model to detect differences between a series of images overtime. In this configuration, the tracking system is configured to usethe machine learning model to determine whether an item has been removedor replaced from a rack. After detecting that an item has been removedor replaces from a rack, the tracking system may then identify the itemand modify a digital cart of a person that is adjacent to the rack basedon the identified item. This configuration is described in more detailusing FIGS. 43-44 .

In one embodiment, the tracking system is configured to detect when aperson is interacting with a self-serve beverage machine and to trackthe beverages that are obtained by the person. In this configuration,the tracking system may employ one or more zones that are used toautomatically detect the type and size of beverages that a person isretrieving from the self-serve beverage machine. After determining thetype and size of beverages that a person retrieves, the tracking systemthen modifies a digital cart of the person based on the identifiedbeverages. This configuration is described in more detail using FIGS.45-47 .

In one embodiment, the tracking system is configured to employ a sensormounting system with adjustable camera positions. The sensor mountingsystem generally includes a sensor, a mounting ring, a faceplatesupport, and a faceplate. The mounting ring includes a first opening anda first plurality of threads that are disposed on an interior surface ofthe first opening. The faceplate support is disposed within the firstopening of the mounting ring. The faceplate support includes a secondplurality of threads that are configured to engage the first pluralityof threads of the mounting ring and a second opening. The faceplate isdisposed within the second opening of the faceplate support. Thefaceplate is coupled to the sensor and is configured to rotate withinthe second opening of the faceplate support. This configuration isdescribed in more detail using FIGS. 48-56 .

In one embodiment, the tracking system is configured to using distancemeasuring devices (e.g. draw wire encoders) to generate a homography fora sensor. In this configuration, a platform that comprises one or moremarkers is repositioned within the field of view of a sensor. Thetracking system is configured to obtain location information for theplatform and the markers from the distance measuring devices while theplatform is repositioned within a space. The tracking system thencomputes a homography for the sensor based on the location informationfrom the distance measuring device and the pixel locations of themarkers within a frame captured by the sensor. This configuration isdescribed in more detail using FIGS. 57-59 .

In one embodiment, the tracking system is configured to define a zonewithin a frame from a sensor using a region-of-interest (ROI) marker. Inthis configuration, the tracking system uses the ROI marker to define azone within frames from a sensor. The tracking system then uses thedefined zone to reduce the search space when performing object detectionto determine whether a person is removing or replacing an item from afood rack. This configuration is described in more detail using FIGS.60-64 .

In one embodiment, the tracking system is configured to update ahomography for a sensor in response to determining that the sensor hasmoved since its homography was first computed. In this configuration,the tracking system determines translation coefficients and/or rotationcoefficients and updates the homography for the sensor by applying thetranslation coefficients and/or rotation coefficients. Thisconfiguration is described in more detail using FIGS. 65-67 .

In one embodiment, the tracking system is configured to detect andcorrect homography errors based on the location of a sensor. In thisconfiguration, the tracking system determines an error between anestimated location of a sensor using a homography and the actuallocation of the sensor. The tracking system is configured to recomputethe homography for the sensor in response to determining that the erroris beyond the accuracy tolerances of the system. This configuration isdescribed in more detail using FIGS. 69 and 70 .

In one embodiment, the tracking system is configured to detect andcorrect homography errors using distances between markers. In thisconfiguration, the tracking system determines whether a distancemeasurement error that is computed using a homography exceeds theaccuracy tolerances of the system. The tracking system is configured torecompute the homography for a sensor in response to determining thatthe distance measurement error is beyond the accuracy tolerances of thesystem. This configuration is described in more detail using FIGS. 70and 71 .

In one embodiment, the tracking system is configured to detect andcorrect homography errors using a disparity mapping between adjacentsensors. In this configuration, the tracking system determines whether apixel location that is computed using a homography is within theaccuracy tolerances of the system. The tracking system is configured torecompute the homography in response to determining the results of usingthe homography are beyond the accuracy tolerances of the system. Thisconfiguration is described in more detail using FIGS. 72 and 73 .

In one embodiment, the tracking system is configured to detect andcorrect homography errors using adjacent sensors. In this configuration,the tracking system determines whether a distance measurement error thatis computed using adjacent sensors exceeds the accuracy tolerances ofthe system. The tracking system is configured to recompute thehomographies for the sensors in response to determining that thedistance measurement error is beyond the accuracy tolerances of thesystem. This configuration is described in more detail using FIGS. 74and 75 .

Certain embodiments of the present disclosure may include some, all, ornone of these advantages. These advantages and other features will bemore clearly understood from the following detailed description taken inconjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure, reference is nowmade to the following brief description, taken in connection with theaccompanying drawings and detailed description, wherein like referencenumerals represent like parts.

FIG. 1 is a schematic diagram of an embodiment of a tracking systemconfigured to track objects within a space;

FIG. 2 is a flowchart of an embodiment of a sensor mapping method forthe tracking system;

FIG. 3 is an example of a sensor mapping process for the trackingsystem;

FIG. 4 is an example of a frame from a sensor in the tracking system;

FIG. 5A is an example of a sensor mapping for a sensor in the trackingsystem;

FIG. 5B is another example of a sensor mapping for a sensor in thetracking system;

FIG. 6 is a flowchart of an embodiment of a sensor mapping method forthe tracking system using a marker grid;

FIG. 7 is an example of a sensor mapping process for the tracking systemusing a marker grid;

FIG. 8 is a flowchart of an embodiment of a shelf position calibrationmethod for the tracking system;

FIG. 9 is an example of a shelf position calibration process for thetracking system;

FIG. 10 is a flowchart of an embodiment of a tracking hand off methodfor the tracking system;

FIG. 11 is an example of a tracking hand off process for the trackingsystem;

FIG. 12 is a flowchart of an embodiment of a shelf interaction detectionmethod for the tracking system;

FIG. 13 is a front view of an example of a shelf interaction detectionprocess for the tracking system;

FIG. 14 is an overhead view of an example of a shelf interactiondetection process for the tracking system;

FIG. 15 is a flowchart of an embodiment of an item assigning method forthe tracking system;

FIG. 16 is a flowchart of an embodiment of an item identification methodfor the tracking system;

FIG. 17 is a flowchart of an embodiment of a misplaced itemidentification method for the tracking system;

FIG. 18 is an example of an item identification process for the trackingsystem;

FIG. 19 is a diagram illustrating the determination and use ofauto-exclusion zones by the tracking system;

FIG. 20 is an example auto-exclusion zone map generated by the trackingsystem;

FIG. 21 is a flowchart illustrating an example method of generating andusing auto-exclusion zones for object tracking using the trackingsystem;

FIG. 22 is a diagram illustrating the detection of closely spacedobjects using the tracking system;

FIG. 23 is a flowchart illustrating an example method of detectingclosely spaced objects using the tracking system;

FIGS. 24A-C are diagrams illustrating the tracking of a person in localimage frames and in the global plane of space 102 using the trackingsystem;

FIGS. 25A-B illustrate the implementation of a particle filter trackerby the tracking system;

FIG. 26 is a flow diagram illustrating an example method of local andglobal object tracking using the tracking system;

FIG. 27 is a diagram illustrating the use of candidate lists for objectidentification during object tracking by the tracking system;

FIG. 28 is a flowchart illustrating an example method of maintainingcandidate lists during object tracking by the tracking system;

FIG. 29 is a diagram illustrating an example tracking subsystem for usein the tracking system;

FIG. 30 is a diagram illustrating the determination of descriptors basedon object features using the tracking system;

FIGS. 31A-C are diagrams illustrating the use of descriptors forre-identification during object tracking by the tracking system;

FIG. 32 is a flowchart illustrating an example method of objectre-identification during object tracking using the tracking system;

FIGS. 33A-C are diagrams illustrating the assignment of an item to aperson using the tracking system;

FIG. 34 is a flowchart of an example method for assigning an item to aperson using the tracking system;

FIG. 35 is a flowchart of an example method of contour dilation-baseditem assignment using the tracking system;

FIGS. 36A-B are diagrams illustrating item tracking-based itemassignment using the tracking system;

FIG. 37 is a flowchart of an example method of item tracking-based itemassignment using the tracking system;

FIG. 38 is an embodiment of a device configured to track objects withina space;

FIG. 39 is an example of using an angled-view sensor to assign aselected item to a person moving about the space;

FIG. 40 is a flow diagram of an embodiment of a triggering event baseditem assignment process;

FIGS. 41 and 42 are an example of performing object-based detectionbased on a wrist-area region-of-interest;

FIG. 43 is a flow diagram of an embodiment of a process for determiningwhether an item is being replaced or removed from a rack;

FIG. 44 is an embodiment of an item assignment method for the trackingsystem;

FIG. 45 is a schematic diagram of an embodiment of a self-serve beverageassignment system;

FIG. 46 is a flow chart of an embodiment of a beverage assignment methodusing the tracking system;

FIG. 47 is a flow chart of another embodiment of a beverage assignmentmethod using the tracking system;

FIG. 48 is a perspective view of an embodiment of a faceplate supportbeing installed into a mounting ring;

FIG. 49 is a perspective view of an embodiment of a mounting ring;

FIG. 50 is a perspective view of an embodiment of a faceplate support;

FIG. 51 is a perspective view of an embodiment of a faceplate;

FIG. 52 is a perspective view of an embodiment of a sensor installedonto a faceplate;

FIG. 53 is a bottom perspective view of an embodiment of a sensorinstalled onto a faceplate;

FIG. 54 is a perspective view of another embodiment of a sensorinstalled onto a faceplate;

FIG. 55 is a bottom perspective view of an embodiment of a sensorinstalled onto a faceplate;

FIG. 56 is a perspective view of an embodiment of a sensor assemblyinstalled onto an adjustable positioning system;

FIG. 57 is an overhead view of an example of a draw wire encoder system;

FIG. 58 is a perspective view of a platform for a draw wire encodersystem;

FIG. 59 is a flowchart of an embodiment of a sensor mapping processusing a draw wire encoder system;

FIG. 60 is a flowchart of an object tracking process for the trackingsystem;

FIG. 61 is an example of a first phase of an object tracking process forthe tracking system;

FIG. 62 is an example of an affine transformation for the objecttracking process;

FIG. 63 is an example of a second phase of the object tracking processfor the tracking system;

FIG. 64 is an example of a binary mask for the object tracking process;

FIG. 65 is a flowchart of an embodiment of a sensor reconfigurationprocess for the tracking system;

FIG. 66 is an overhead view of an example of the sensor reconfigurationprocess for the tracking system;

FIG. 67 is an example of applying a transformation matrix to ahomography matrix to update the homography matrix;

FIG. 68 is a flowchart of an embodiment of a homography error correctionprocess for the tracking system;

FIG. 69 is an example of a homography error correction process for thetracking system;

FIG. 70 is a flowchart of another embodiment of a homography errorcorrection process for the tracking system;

FIG. 71 is another example of a homography error correction process forthe tracking system;

FIG. 72 is a flowchart of another embodiment of a homography errorcorrection process for the tracking system;

FIG. 73 is another example of a homography error correction process forthe tracking system;

FIG. 74 is a flowchart of another embodiment of a homography errorcorrection process for the tracking system; and

FIG. 75 is another example of a homography error correction process forthe tracking system.

DETAILED DESCRIPTION

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which person in the store removed the itemand to charge that person for the item without needing to ring up theitem at a register. In other words, the person can walk into the store,take items, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. Increasing the number of cameras poses atechnical challenge because each camera only provides a field of viewfor a portion of the physical space. This means that information fromeach camera needs to be processed independently to identify and trackpeople and objects within the field of view of a particular camera. Theinformation from each camera then needs to be combined and processed asa collective in order to track people and objects within the physicalspace.

Additional information is disclosed in U.S. patent application Ser. No.16/663,633 entitled, “Scalable Position Tracking System For TrackingPosition In Large Spaces” and U.S. patent application Ser. No.16/664,470 entitled, “Customer-Based Video Feed” which are both herebyincorporated by reference herein as if reproduced in their entirety.

Tracking System Overview

FIG. 1 is a schematic diagram of an embodiment of a tracking system 100that is configured to track objects within a space 102. As discussedabove, the tracking system 100 may be installed in a space 102 (e.g. astore) so that shoppers need not engage in the conventional checkoutprocess. Although the example of a store is used in this disclosure,this disclosure contemplates that the tracking system 100 may beinstalled and used in any type of physical space (e.g. a room, anoffice, an outdoor stand, a mall, a supermarket, a convenience store, apop-up store, a warehouse, a storage center, an amusement park, anairport, an office building, etc.). Generally, the tracking system 100(or components thereof) is used to track the positions of people and/orobjects within these spaces 102 for any suitable purpose. For example,at an airport, the tracking system 100 can track the positions oftravelers and employees for security purposes. As another example, at anamusement park, the tracking system 100 can track the positions of parkguests to gauge the popularity of attractions. As yet another example,at an office building, the tracking system 100 can track the positionsof employees and staff to monitor their productivity levels.

In FIG. 1 , the space 102 is a store that comprises a plurality of itemsthat are available for purchase. The tracking system 100 may beinstalled in the store so that shoppers need not engage in theconventional checkout process to purchase items from the store. In thisexample, the store may be a convenience store or a grocery store. Inother examples, the store may not be a physical building, but a physicalspace or environment where shoppers may shop. For example, the store maybe a grab and go pantry at an airport, a kiosk in an office building, anoutdoor market at a park, etc.

In FIG. 1 , the space 102 comprises one or more racks 112. Each rack 112comprises one or more shelves that are configured to hold and displayitems. In some embodiments, the space 102 may comprise refrigerators,coolers, freezers, or any other suitable type of furniture for holdingor displaying items for purchase. The space 102 may be configured asshown or in any other suitable configuration.

In this example, the space 102 is a physical structure that includes anentryway through which shoppers can enter and exit the space 102. Thespace 102 comprises an entrance area 114 and an exit area 116. In someembodiments, the entrance area 114 and the exit area 116 may overlap orare the same area within the space 102. The entrance area 114 isadjacent to an entrance (e.g. a door) of the space 102 where a personenters the space 102. In some embodiments, the entrance area 114 maycomprise a turnstile or gate that controls the flow of traffic into thespace 102. For example, the entrance area 114 may comprise a turnstilethat only allows one person to enter the space 102 at a time. Theentrance area 114 may be adjacent to one or more devices (e.g. sensors108 or a scanner 115) that identify a person as they enter space 102. Asan example, a sensor 108 may capture one or more images of a person asthey enter the space 102. As another example, a person may identifythemselves using a scanner 115. Examples of scanners 115 include, butare not limited to, a QR code scanner, a barcode scanner, a near-fieldcommunication (NFC) scanner, or any other suitable type of scanner thatcan receive an electronic code embedded with information that uniquelyidentifies a person. For instance, a shopper may scan a personal device(e.g. a smart phone) on a scanner 115 to enter the store. When theshopper scans their personal device on the scanner 115, the personaldevice may provide the scanner 115 with an electronic code that uniquelyidentifies the shopper. After the shopper is identified and/orauthenticated, the shopper is allowed to enter the store. In oneembodiment, each shopper may have a registered account with the store toreceive an identification code for the personal device.

After entering the space 102, the shopper may move around the interiorof the store. As the shopper moves throughout the space 102, the shoppermay shop for items by removing items from the racks 112. The shopper canremove multiple items from the racks 112 in the store to purchase thoseitems. When the shopper has finished shopping, the shopper may leave thestore via the exit area 116. The exit area 116 is adjacent to an exit(e.g. a door) of the space 102 where a person leaves the space 102. Insome embodiments, the exit area 116 may comprise a turnstile or gatethat controls the flow of traffic out of the space 102. For example, theexit area 116 may comprise a turnstile that only allows one person toleave the space 102 at a time. In some embodiments, the exit area 116may be adjacent to one or more devices (e.g. sensors 108 or a scanner115) that identify a person as they leave the space 102. For example, ashopper may scan their personal device on the scanner 115 before aturnstile or gate will open to allow the shopper to exit the store. Whenthe shopper scans their personal device on the scanner 115, the personaldevice may provide an electronic code that uniquely identifies theshopper to indicate that the shopper is leaving the store. When theshopper leaves the store, an account for the shopper is charged for theitems that the shopper removed from the store. Through this process thetracking system 100 allows the shopper to leave the store with theiritems without engaging in a conventional checkout process.

Global Plane Overview

In order to describe the physical location of people and objects withinthe space 102, a global plane 104 is defined for the space 102. Theglobal plane 104 is a user-defined coordinate system that is used by thetracking system 100 to identify the locations of objects within aphysical domain (i.e. the space 102). Referring to FIG. 1 as an example,a global plane 104 is defined such that an x-axis and a y-axis areparallel with a floor of the space 102. In this example, the z-axis ofthe global plane 104 is perpendicular to the floor of the space 102. Alocation in the space 102 is defined as a reference location 101 ororigin for the global plane 104. In FIG. 1 , the global plane 104 isdefined such that reference location 101 corresponds with a corner ofthe store. In other examples, the reference location 101 may be locatedat any other suitable location within the space 102.

In this configuration, physical locations within the space 102 can bedescribed using (x,y) coordinates in the global plane 104. As anexample, the global plane 104 may be defined such that one unit in theglobal plane 104 corresponds with one meter in the space 102. In otherwords, an x-value of one in the global plane 104 corresponds with anoffset of one meter from the reference location 101 in the space 102. Inthis example, a person that is standing in the corner of the space 102at the reference location 101 will have an (x,y) coordinate with a valueof (0,0) in the global plane 104. If person moves two meters in thepositive x-axis direction and two meters in the positive y-axisdirection, then their new (x,y) coordinate will have a value of (2,2).In other examples, the global plane 104 may be expressed using inches,feet, or any other suitable measurement units.

Once the global plane 104 is defined for the space 102, the trackingsystem 100 uses (x,y) coordinates of the global plane 104 to track thelocation of people and objects within the space 102. For example, as ashopper moves within the interior of the store, the tracking system 100may track their current physical location within the store using (x,y)coordinates of the global plane 104.

Tracking System Hardware

In one embodiment, the tracking system 100 comprises one or more clients105, one or more servers 106, one or more scanners 115, one or moresensors 108, and one or more weight sensors 110. The one or more clients105, one or more servers 106, one or more scanners 115, one or moresensors 108, and one or more weight sensors 110 may be in signalcommunication with each other over a network 107. The network 107 may beany suitable type of wireless and/or wired network including, but notlimited to, all or a portion of the Internet, an Intranet, a Bluetoothnetwork, a WIFI network, a Zigbee network, a Z-wave network, a privatenetwork, a public network, a peer-to-peer network, the public switchedtelephone network, a cellular network, a local area network (LAN), ametropolitan area network (MAN), a wide area network (WAN), and asatellite network. The network 107 may be configured to support anysuitable type of communication protocol as would be appreciated by oneof ordinary skill in the art. The tracking system 100 may be configuredas shown or in any other suitable configuration.

Sensors

The tracking system 100 is configured to use sensors 108 to identify andtrack the location of people and objects within the space 102. Forexample, the tracking system 100 uses sensors 108 to capture images orvideos of a shopper as they move within the store. The tracking system100 may process the images or videos provided by the sensors 108 toidentify the shopper, the location of the shopper, and/or any items thatthe shopper picks up.

Examples of sensors 108 include, but are not limited to, cameras, videocameras, web cameras, printed circuit board (PCB) cameras, depth sensingcameras, time-of-flight cameras, LiDARs, structured light cameras, orany other suitable type of imaging device.

Each sensor 108 is positioned above at least a portion of the space 102and is configured to capture overhead view images or videos of at leasta portion of the space 102. In one embodiment, the sensors 108 aregenerally configured to produce videos of portions of the interior ofthe space 102. These videos may include frames or images 302 of shopperswithin the space 102. Each frame 302 is a snapshot of the people and/orobjects within the field of view of a particular sensor 108 at aparticular moment in time. A frame 302 may be a two-dimensional (2D)image or a three-dimensional (3D) image (e.g. a point cloud or a depthmap). In this configuration, each frame 302 is of a portion of a globalplane 104 for the space 102. Referring to FIG. 4 as an example, a frame302 comprises a plurality of pixels that are each associated with apixel location 402 within the frame 302. The tracking system 100 usespixel locations 402 to describe the location of an object with respectto pixels in a frame 302 from a sensor 108. In the example shown in FIG.4 , the tracking system 100 can identify the location of differentmarker 304 within the frame 302 using their respective pixel locations402. The pixel location 402 corresponds with a pixel row and a pixelcolumn where a pixel is located within the frame 302. In one embodiment,each pixel is also associated with a pixel value 404 that indicates adepth or distance measurement in the global plane 104. For example, apixel value 404 may correspond with a distance between a sensor 108 anda surface in the space 102.

Each sensor 108 has a limited field of view within the space 102. Thismeans that each sensor 108 may only be able to capture a portion of thespace 102 within their field of view. To provide complete coverage ofthe space 102, the tracking system 100 may use multiple sensors 108configured as a sensor array. In FIG. 1 , the sensors 108 are configuredas a three by four sensor array. In other examples, a sensor array maycomprise any other suitable number and/or configuration of sensors 108.In one embodiment, the sensor array is positioned parallel with thefloor of the space 102. In some embodiments, the sensor array isconfigured such that adjacent sensors 108 have at least partiallyoverlapping fields of view. In this configuration, each sensor 108captures images or frames 302 of a different portion of the space 102which allows the tracking system 100 to monitor the entire space 102 bycombining information from frames 302 of multiple sensors 108. Thetracking system 100 is configured to map pixel locations 402 within eachsensor 108 to physical locations in the space 102 using homographies118. A homography 118 is configured to translate between pixel locations402 in a frame 302 captured by a sensor 108 and (x,y) coordinates in theglobal plane 104 (i.e. physical locations in the space 102). Thetracking system 100 uses homographies 118 to correlate between a pixellocation 402 in a particular sensor 108 with a physical location in thespace 102. In other words, the tracking system 100 uses homographies 118to determine where a person is physically located in the space 102 basedon their pixel location 402 within a frame 302 from a sensor 108. Sincethe tracking system 100 uses multiple sensors 108 to monitor the entirespace 102, each sensor 108 is uniquely associated with a differenthomography 118 based on the sensor's 108 physical location within thespace 102. This configuration allows the tracking system 100 todetermine where a person is physically located within the entire space102 based on which sensor 108 they appear in and their location within aframe 302 captured by that sensor 108. Additional information abouthomographies 118 is described in FIGS. 2-7 .

Weight Sensors

The tracking system 100 is configured to use weight sensors 110 todetect and identify items that a person picks up within the space 102.For example, the tracking system 100 uses weight sensors 110 that arelocated on the shelves of a rack 112 to detect when a shopper removes anitem from the rack 112. Each weight sensor 110 may be associated with aparticular item which allows the tracking system 100 to identify whichitem the shopper picked up.

A weight sensor 110 is generally configured to measure the weight ofobjects (e.g. products) that are placed on or near the weight sensor110. For example, a weight sensor 110 may comprise a transducer thatconverts an input mechanical force (e.g. weight, tension, compression,pressure, or torque) into an output electrical signal (e.g. current orvoltage). As the input force increases, the output electrical signal mayincrease proportionally. The tracking system 100 is configured toanalyze the output electrical signal to determine an overall weight forthe items on the weight sensor 110.

Examples of weight sensors 110 include, but are not limited to, apiezoelectric load cell or a pressure sensor. For example, a weightsensor 110 may comprise one or more load cells that are configured tocommunicate electrical signals that indicate a weight experienced by theload cells. For instance, the load cells may produce an electricalcurrent that varies depending on the weight or force experienced by theload cells. The load cells are configured to communicate the producedelectrical signals to a server 105 and/or a client 106 for processing.

Weight sensors 110 may be positioned onto furniture (e.g. racks 112)within the space 102 to hold one or more items. For example, one or moreweight sensors 110 may be positioned on a shelf of a rack 112. Asanother example, one or more weight sensors 110 may be positioned on ashelf of a refrigerator or a cooler. As another example, one or moreweight sensors 110 may be integrated with a shelf of a rack 112. Inother examples, weight sensors 110 may be positioned in any othersuitable location within the space 102.

In one embodiment, a weight sensor 110 may be associated with aparticular item. For instance, a weight sensor 110 may be configured tohold one or more of a particular item and to measure a combined weightfor the items on the weight sensor 110. When an item is picked up fromthe weight sensor 110, the weight sensor 110 is configured to detect aweight decrease. In this example, the weight sensor 110 is configured touse stored information about the weight of the item to determine anumber of items that were removed from the weight sensor 110. Forexample, a weight sensor 110 may be associated with an item that has anindividual weight of eight ounces. When the weight sensor 110 detects aweight decrease of twenty-four ounces, the weight sensor 110 maydetermine that three of the items were removed from the weight sensor110. The weight sensor 110 is also configured to detect a weightincrease when an item is added to the weight sensor 110. For example, ifan item is returned to the weight sensor 110, then the weight sensor 110will determine a weight increase that corresponds with the individualweight for the item associated with the weight sensor 110.

Servers

A server 106 may be formed by one or more physical devices configured toprovide services and resources (e.g. data and/or hardware resources) forthe tracking system 100. Additional information about the hardwareconfiguration of a server 106 is described in FIG. 38 . In oneembodiment, a server 106 may be operably coupled to one or more sensors108 and/or weight sensors 110. The tracking system 100 may comprise anysuitable number of servers 106. For example, the tracking system 100 maycomprise a first server 106 that is in signal communication with a firstplurality of sensors 108 in a sensor array and a second server 106 thatis in signal communication with a second plurality of sensors 108 in thesensor array. As another example, the tracking system 100 may comprise afirst server 106 that is in signal communication with a plurality ofsensors 108 and a second server 106 that is in signal communication witha plurality of weight sensors 110. In other examples, the trackingsystem 100 may comprise any other suitable number of servers 106 thatare each in signal communication with one or more sensors 108 and/orweight sensors 110.

A server 106 may be configured to process data (e.g. frames 302 and/orvideo) for one or more sensors 108 and/or weight sensors 110. In oneembodiment, a server 106 may be configured to generate homographies 118for sensors 108. As discussed above, the generated homographies 118allow the tracking system 100 to determine where a person is physicallylocated within the entire space 102 based on which sensor 108 theyappear in and their location within a frame 302 captured by that sensor108. In this configuration, the server 106 determines coefficients for ahomography 118 based on the physical location of markers in the globalplane 104 and the pixel locations of the markers in an image from asensor 108. Examples of the server 106 performing this process aredescribed in FIGS. 2-7 .

In one embodiment, a server 106 is configured to calibrate a shelfposition within the global plane 104 using sensors 108. This processallows the tracking system 100 to detect when a rack 112 or sensor 108has moved from its original location within the space 102. In thisconfiguration, the server 106 periodically compares the current shelflocation of a rack 112 to an expected shelf location for the rack 112using a sensor 108. In the event that the current shelf location doesnot match the expected shelf location, then the server 106 will use oneor more other sensors 108 to determine whether the rack 112 has moved orwhether the first sensor 108 has moved. An example of the server 106performing this process is described in FIGS. 8 and 9 .

In one embodiment, a server 106 is configured to hand off trackinginformation for an object (e.g. a person) as it moves between the fieldsof view of adjacent sensors 108. This process allows the tracking system100 to track people as they move within the interior of the space 102.In this configuration, the server 106 tracks an object's movement withinthe field of view of a first sensor 108 and then hands off trackinginformation (e.g. an object identifier) for the object as it enters thefield of view of a second adjacent sensor 108. An example of the server106 performing this process is described in FIGS. 10 and 11 .

In one embodiment, a server 106 is configured to detect shelfinteractions using a virtual curtain. This process allows the trackingsystem 100 to identify items that a person picks up from a rack 112. Inthis configuration, the server 106 is configured to process an imagecaptured by a sensor 108 to determine where a person is interacting witha shelf of a rack 112. The server 106 uses a predetermined zone withinthe image as a virtual curtain that is used to determine which regionand which shelf of a rack 112 that a person is interacting with. Anexample of the server 106 performing this process is described in FIGS.12-14 .

In one embodiment, a server 106 is configured to detect when an item hasbeen picked up from a rack 112 and to determine which person to assignthe item to using a predefined zone that is associated with the rack112. This process allows the tracking system 100 to associate items on arack 112 with the person that picked up the item. In this configuration,the server 106 detects that an item has been picked up using a weightsensor 110. The server 106 then uses a sensor 108 to identify a personwithin a predefined zone that is associated with the rack 112. Once theitem and the person have been identified, the server 106 will add theitem to a digital cart that is associated with the identified person. Anexample of the server 106 performing this process is described in FIGS.15 and 18 .

In one embodiment, a server 106 is configured to identify an object thathas a non-uniform weight and to assign the item to a person's digitalcart. This process allows the tracking system 100 to identify items thata person picks up that cannot be identified based on just their weight.For example, the weight of fresh food is not constant and will vary fromitem to item. In this configuration, the server 106 uses a sensor 108 toidentify markers (e.g. text or symbols) on an item that has been pickedup. The server 106 uses the identified markers to then identify whichitem was picked up. The server 106 then uses the sensor 108 to identifya person within a predefined zone that is associated with the rack 112.Once the item and the person have been identified, the server 106 willadd the item to a digital cart that is associated with the identifiedperson. An example of the server 106 performing this process isdescribed in FIGS. 16 and 18 .

In one embodiment, a server 106 is configured to identify items thathave been misplaced on a rack 112. This process allows the trackingsystem 100 to remove items from a shopper's digital cart when theshopper puts down an item regardless of whether they put the item backin its proper location. For example, a person may put back an item inthe wrong location on the rack 112 or on the wrong rack 112. In thisconfiguration, the server 106 uses a weight sensor 110 to detect that anitem has been put back on rack 112 and to determine that the item is notin the correct location based on its weight. The server 106 then uses asensor 108 to identify the person that put the item on the rack 112 andanalyzes their digital cart to determine which item they put back basedon the weights of the items in their digital cart. An example of theserver 106 performing this process is described in FIGS. 17 and 18 .

Clients

In some embodiments, one or more sensors 108 and/or weight sensors 110are operably coupled to a server 106 via a client 105. In oneembodiment, the tracking system 100 comprises a plurality of clients 105that may each be operably coupled to one or more sensors 108 and/orweight sensors 110. For example, first client 105 may be operablycoupled to one or more sensors 108 and/or weight sensors 110 and asecond client 105 may be operably coupled to one or more other sensors108 and/or weight sensors 110. A client 105 may be formed by one or morephysical devices configured to process data (e.g. frames 302 and/orvideo) for one or more sensors 108 and/or weight sensors 110. A client105 may act as an intermediary for exchanging data between a server 106and one or more sensors 108 and/or weight sensors 110. The combinationof one or more clients 105 and a server 106 may also be referred to as atracking subsystem. In this configuration, a client 105 may beconfigured to provide image processing capabilities for images or frames302 that are captured by a sensor 108. The client 105 is furtherconfigured to send images, processed images, or any other suitable typeof data to the server 106 for further processing and analysis. In someembodiments, a client 105 may be configured to perform one or more ofthe processes described above for the server 106.

Sensor Mapping Process

FIG. 2 is a flowchart of an embodiment of a sensor mapping method 200for the tracking system 100. The tracking system 100 may employ method200 to generate a homography 118 for a sensor 108. As discussed above, ahomography 118 allows the tracking system 100 to determine where aperson is physically located within the entire space 102 based on whichsensor 108 they appear in and their location within a frame 302 capturedby that sensor 108. Once generated, the homography 118 can be used totranslate between pixel locations 402 in images (e.g. frames 302)captured by a sensor 108 and (x,y) coordinates 306 in the global plane104 (i.e. physical locations in the space 102). The following is anon-limiting example of the process for generating a homography 118 fora single sensor 108. This same process can be repeated for generating ahomography 118 for other sensors 108.

At step 202, the tracking system 100 receives (x,y) coordinates 306 formarkers 304 in the space 102. Referring to FIG. 3 as an example, eachmarker 304 is an object that identifies a known physical location withinthe space 102. The markers 304 are used to demarcate locations in thephysical domain (i.e. the global plane 104) that can be mapped to pixellocations 402 in a frame 302 from a sensor 108. In this example, themarkers 304 are represented as stars on the floor of the space 102. Amarker 304 may be formed of any suitable object that can be observed bya sensor 108. For example, a marker 304 may be tape or a sticker that isplaced on the floor of the space 102. As another example, a marker 304may be a design or marking on the floor of the space 102. In otherexamples, markers 304 may be positioned in any other suitable locationwithin the space 102 that is observable by a sensor 108. For instance,one or more markers 304 may be positioned on top of a rack 112.

In one embodiment, the (x,y) coordinates 306 for markers 304 areprovided by an operator. For example, an operator may manually placemarkers 304 on the floor of the space 102. The operator may determine an(x,y) location 306 for a marker 304 by measuring the distance betweenthe marker 304 and the reference location 101 for the global plane 104.The operator may then provide the determined (x,y) location 306 to aserver 106 or a client 105 of the tracking system 100 as an input.

Referring to the example in FIG. 3 , the tracking system 100 may receivea first (x,y) coordinate 306A for a first marker 304A in a space 102 anda second (x,y) coordinate 306B for a second marker 304B in the space102. The first (x,y) coordinate 306A describes the physical location ofthe first marker 304A with respect to the global plane 104 of the space102. The second (x,y) coordinate 306B describes the physical location ofthe second marker 304B with respect to the global plane 104 of the space102. The tracking system 100 may repeat the process of obtaining (x,y)coordinates 306 for any suitable number of additional markers 304 withinthe space 102.

Once the tracking system 100 knows the physical location of the markers304 within the space 102, the tracking system 100 then determines wherethe markers 304 are located with respect to the pixels in the frame 302of a sensor 108. Returning to FIG. 2 at step 204, the tracking system100 receives a frame 302 from a sensor 108. Referring to FIG. 4 as anexample, the sensor 108 captures an image or frame 302 of the globalplane 104 for at least a portion of the space 102. In this example, theframe 302 comprises a plurality of markers 304.

Returning to FIG. 2 at step 206, the tracking system 100 identifiesmarkers 304 within the frame 302 of the sensor 108. In one embodiment,the tracking system 100 uses object detection to identify markers 304within the frame 302. For example, the markers 304 may have knownfeatures (e.g. shape, pattern, color, text, etc.) that the trackingsystem 100 can search for within the frame 302 to identify a marker 304.Referring to the example in FIG. 3 , each marker 304 has a star shape.In this example, the tracking system 100 may search the frame 302 forstar shaped objects to identify the markers 304 within the frame 302.The tracking system 100 may identify the first marker 304A, the secondmarker 304B, and any other markers 304 within the frame 302. In otherexamples, the tracking system 100 may use any other suitable featuresfor identifying markers 304 within the frame 302. In other embodiments,the tracking system 100 may employ any other suitable image processingtechnique for identifying markers 302 within the frame 302. For example,the markers 304 may have a known color or pixel value. In this example,the tracking system 100 may use thresholds to identify the markers 304within frame 302 that correspond with the color or pixel value of themarkers 304.

Returning to FIG. 2 at step 208, the tracking system 100 determines thenumber of identified markers 304 within the frame 302. Here, trackingsystem 100 counts the number of markers 304 that were detected withinthe frame 302. Referring to the example in FIG. 3 , the tracking system100 detects eight markers 304 within the frame 302.

Returning to FIG. 2 at step 210, the tracking system 100 determineswhether the number of identified markers 304 is greater than or equal toa predetermined threshold value. In some embodiments, the predeterminedthreshold value is proportional to a level of accuracy for generating ahomography 118 for a sensor 108. Increasing the predetermined thresholdvalue may increase the accuracy when generating a homography 118 whiledecreasing the predetermined threshold value may decrease the accuracywhen generating a homography 118. As an example, the predeterminedthreshold value may be set to a value of six. In the example shown inFIG. 3 , the tracking system 100 identified eight markers 304 which isgreater than the predetermined threshold value. In other examples, thepredetermined threshold value may be set to any other suitable value.The tracking system 100 returns to step 204 in response to determiningthat the number of identified markers 304 is less than the predeterminedthreshold value. In this case, the tracking system 100 returns to step204 to capture another frame 302 of the space 102 using the same sensor108 to try to detect more markers 304. Here, the tracking system 100tries to obtain a new frame 302 that includes a number of markers 304that is greater than or equal to the predetermined threshold value. Forexample, the tracking system 100 may receive new frame 302 of the space102 after an operator adds one or more additional markers 304 to thespace 102. As another example, the tracking system 100 may receive newframe 302 after lighting conditions have been changed to improve thedetectability of the markers 304 within the frame 302. In otherexamples, the tracking system 100 may receive new frame 302 after anykind of change that improves the detectability of the markers 304 withinthe frame 302.

The tracking system 100 proceeds to step 212 in response to determiningthat the number of identified markers 304 is greater than or equal tothe predetermined threshold value. At step 212, the tracking system 100determines pixel locations 402 in the frame 302 for the identifiedmarkers 304. For example, the tracking system 100 determines a firstpixel location 402A within the frame 302 that corresponds with the firstmarker 304A and a second pixel location 402B within the frame 302 thatcorresponds with the second marker 304B. The first pixel location 402Acomprises a first pixel row and a first pixel column indicating wherethe first marker 304A is located in the frame 302. The second pixellocation 402B comprises a second pixel row and a second pixel columnindicating where the second marker 304B is located in the frame 302.

At step 214, the tracking system 100 generates a homography 118 for thesensor 108 based on the pixel locations 402 of identified markers 304with the frame 302 of the sensor 108 and the (x,y) coordinate 306 of theidentified markers 304 in the global plane 104. In one embodiment, thetracking system 100 correlates the pixel location 402 for each of theidentified markers 304 with its corresponding (x,y) coordinate 306.Continuing with the example in FIG. 3 , the tracking system 100associates the first pixel location 402A for the first marker 304A withthe first (x,y) coordinate 306A for the first marker 304A. The trackingsystem 100 also associates the second pixel location 402B for the secondmarker 304B with the second (x,y) coordinate 306B for the second marker304B. The tracking system 100 may repeat the process of associatingpixel locations 402 and (x,y) coordinates 306 for all of the identifiedmarkers 304.

The tracking system 100 then determines a relationship between the pixellocations 402 of identified markers 304 with the frame 302 of the sensor108 and the (x,y) coordinates 306 of the identified markers 304 in theglobal plane 104 to generate a homography 118 for the sensor 108. Thegenerated homography 118 allows the tracking system 100 to map pixellocations 402 in a frame 302 from the sensor 108 to (x,y) coordinates306 in the global plane 104. Additional information about a homography118 is described in FIGS. 5A and 5B. Once the tracking system 100generates the homography 118 for the sensor 108, the tracking system 100stores an association between the sensor 108 and the generatedhomography 118 in memory (e.g. memory 3804).

The tracking system 100 may repeat the process described above togenerate and associate homographies 118 with other sensors 108.Continuing with the example in FIG. 3 , the tracking system 100 mayreceive a second frame 302 from a second sensor 108. In this example,the second frame 302 comprises the first marker 304A and the secondmarker 304B. The tracking system 100 may determine a third pixellocation 402 in the second frame 302 for the first marker 304A, a fourthpixel location 402 in the second frame 302 for the second marker 304B,and pixel locations 402 for any other markers 304. The tracking system100 may then generate a second homography 118 based on the third pixellocation 402 in the second frame 302 for the first marker 304A, thefourth pixel location 402 in the second frame 302 for the second marker304B, the first (x,y) coordinate 306A in the global plane 104 for thefirst marker 304A, the second (x,y) coordinate 306B in the global plane104 for the second marker 304B, and pixel locations 402 and (x,y)coordinates 306 for other markers 304. The second homography 118comprises coefficients that translate between pixel locations 402 in thesecond frame 302 and physical locations (e.g. (x,y) coordinates 306) inthe global plane 104. The coefficients of the second homography 118 aredifferent from the coefficients of the homography 118 that is associatedwith the first sensor 108. This process uniquely associates each sensor108 with a corresponding homography 118 that maps pixel locations 402from the sensor 108 to (x,y) coordinates 306 in the global plane 104.

Homographies

An example of a homography 118 for a sensor 108 is described in FIGS. 5Aand 5B. Referring to FIG. 5A, a homography 118 comprises a plurality ofcoefficients configured to translate between pixel locations 402 in aframe 302 and physical locations (e.g. (x,y) coordinates 306) in theglobal plane 104. In this example, the homography 118 is configured as amatrix and the coefficients of the homography 118 are represented asH₁₁, H₁₂, H₁₃, H₁₄, H₂₁, H₂₂, H₂₃, H₂₄, H₃₁, H₃₂, H₃₃, H₃₄, H₄₁, H₄₂,H₄₃, and H₄₄. The tracking system 100 may generate the homography 118 bydefining a relationship or function between pixel locations 402 in aframe 302 and physical locations (e.g. (x,y) coordinates 306) in theglobal plane 104 using the coefficients. For example, the trackingsystem 100 may define one or more functions using the coefficients andmay perform a regression (e.g. least squares regression) to solve forvalues for the coefficients that project pixel locations 402 of a frame302 of a sensor to (x,y) coordinates 306 in the global plane 104.Referring to the example in FIG. 3 , the homography 118 for the sensor108 is configured to project the first pixel location 402A in the frame302 for the first marker 304A to the first (x,y) coordinate 306A in theglobal plane 104 for the first marker 304A and to project the secondpixel location 402B in the frame 302 for the second marker 304B to thesecond (x,y) coordinate 306B in the global plane 104 for the secondmarker 304B. In other examples, the tracking system 100 may solve forcoefficients of the homography 118 using any other suitable technique.In the example shown in FIG. 5A, the z-value at the pixel location 402may correspond with a pixel value 404. In this case, the homography 118is further configured to translate between pixel values 404 in a frame302 and z-coordinates (e.g. heights or elevations) in the global plane104.

Using Homographies

Once the tracking system 100 generates a homography 118, the trackingsystem 100 may use the homography 118 to determine the location of anobject (e.g. a person) within the space 102 based on the pixel location402 of the object in a frame 302 of a sensor 108. For example, thetracking system 100 may perform matrix multiplication between a pixellocation 402 in a first frame 302 and a homography 118 to determine acorresponding (x,y) coordinate 306 in the global plane 104. For example,the tracking system 100 receives a first frame 302 from a sensor 108 anddetermines a first pixel location in the frame 302 for an object in thespace 102. The tracking system 100 may then apply the homography 118that is associated with the sensor 108 to the first pixel location 402of the object to determine a first (x,y) coordinate 306 that identifiesa first x-value and a first y-value in the global plane 104 where theobject is located.

In some instances, the tracking system 100 may use multiple sensors 108to determine the location of the object. Using multiple sensors 108 mayprovide more accuracy when determining where an object is located withinthe space 102. In this case, the tracking system 100 uses homographies118 that are associated with different sensors 108 to determine thelocation of an object within the global plane 104. Continuing with theprevious example, the tracking system 100 may receive a second frame 302from a second sensor 108. The tracking system 100 may determine a secondpixel location 402 in the second frame 302 for the object in the space102. The tracking system 100 may then apply a second homography 118 thatis associated the second sensor 108 to the second pixel location 402 ofthe object to determine a second (x,y) coordinate 306 that identifies asecond x-value and a second y-value in the global plane 104 where theobject is located.

When the first (x,y) coordinate 306 and the second (x,y) coordinate 306are the same, the tracking system 100 may use either the first (x,y)coordinate 306 or the second (x,y) coordinate 306 as the physicallocation of the object within the space 102. The tracking system 100 mayemploy any suitable clustering technique between the first (x,y)coordinate 306 and the second (x,y) coordinate 306 when the first (x,y)coordinate 306 and the second (x,y) coordinate 306 are not the same. Inthis case, the first (x,y) coordinate 306 and the second (x,y)coordinate 306 are different so the tracking system 100 will need todetermine the physical location of the object within the space 102 basedon the first (x,y) location 306 and the second (x,y) location 306. Forexample, the tracking system 100 may generate an average (x,y)coordinate for the object by computing an average between the first(x,y) coordinate 306 and the second (x,y) coordinate 306. As anotherexample, the tracking system 100 may generate a median (x,y) coordinatefor the object by computing a median between the first (x,y) coordinate306 and the second (x,y) coordinate 306. In other examples, the trackingsystem 100 may employ any other suitable technique to resolvedifferences between the first (x,y) coordinate 306 and the second (x,y)coordinate 306.

The tracking system 100 may use the inverse of the homography 118 toproject from (x,y) coordinates 306 in the global plane 104 to pixellocations 402 in a frame 302 of a sensor 108. For example, the trackingsystem 100 receives an (x,y) coordinate 306 in the global plane 104 foran object. The tracking system 100 identifies a homography 118 that isassociated with a sensor 108 where the object is seen. The trackingsystem 100 may then apply the inverse homography 118 to the (x,y)coordinate 306 to determine a pixel location 402 where the object islocated in the frame 302 for the sensor 108. The tracking system 100 maycompute the matrix inverse of the homograph 500 when the homography 118is represented as a matrix. Referring to FIG. 5B as an example, thetracking system 100 may perform matrix multiplication between an (x,y)coordinates 306 in the global plane 104 and the inverse homography 118to determine a corresponding pixel location 402 in the frame 302 for thesensor 108.

Sensor mapping using a marker grid FIG. 6 is a flowchart of anembodiment of a sensor mapping method 600 for the tracking system 100using a marker grid 702. The tracking system 100 may employ method 600to reduce the amount of time it takes to generate a homography 118 for asensor 108. For example, using a marker grid 702 reduces the amount ofsetup time required to generate a homography 118 for a sensor 108.Typically, each marker 304 is placed within a space 102 and the physicallocation of each marker 304 is determined independently. This process isrepeated for each sensor 108 in a sensor array. In contrast, a markergrid 702 is a portable surface that comprises a plurality of markers304. The marker grid 702 may be formed using carpet, fabric, posterboard, foam board, vinyl, paper, wood, or any other suitable type ofmaterial. Each marker 304 is an object that identifies a particularlocation on the marker grid 702. Examples of markers 304 include, butare not limited to, shapes, symbols, and text. The physical locations ofeach marker 304 on the marker grid 702 are known and are stored inmemory (e.g. marker grid information 716). Using a marker grid 702simplifies and speeds up the process of placing and determining thelocation of markers 304 because the marker grid 702 and its markers 304can be quickly repositioned anywhere within the space 102 without havingto individually move markers 304 or add new markers 304 to the space102. Once generated, the homography 118 can be used to translate betweenpixel locations 402 in frame 302 captured by a sensor 108 and (x,y)coordinates 306 in the global plane 104 (i.e. physical locations in thespace 102).

At step 602, the tracking system 100 receives a first (x,y) coordinate306A for a first corner 704 of a marker grid 702 in a space 102.Referring to FIG. 7 as an example, the marker grid 702 is configured tobe positioned on a surface (e.g. the floor) within the space 102 that isobservable by one or more sensors 108. In this example, the trackingsystem 100 receives a first (x,y) coordinate 306A in the global plane104 for a first corner 704 of the marker grid 702. The first (x,y)coordinate 306A describes the physical location of the first corner 704with respect to the global plane 104. In one embodiment, the first (x,y)coordinate 306A is based on a physical measurement of a distance betweena reference location 101 in the space 102 and the first corner 704. Forexample, the first (x,y) coordinate 306A for the first corner 704 of themarker grid 702 may be provided by an operator. In this example, anoperator may manually place the marker grid 702 on the floor of thespace 102. The operator may determine an (x,y) location 306 for thefirst corner 704 of the marker grid 702 by measuring the distancebetween the first corner 704 of the marker grid 702 and the referencelocation 101 for the global plane 104. The operator may then provide thedetermined (x,y) location 306 to a server 106 or a client 105 of thetracking system 100 as an input.

In another embodiment, the tracking system 100 may receive a signal froma beacon located at the first corner 704 of the marker grid 702 thatidentifies the first (x,y) coordinate 306A. An example of a beaconincludes, but is not limited to, a Bluetooth beacon. For example, thetracking system 100 may communicate with the beacon and determine thefirst (x,y) coordinate 306A based on the time-of-flight of a signal thatis communicated between the tracking system 100 and the beacon. In otherembodiments, the tracking system 100 may obtain the first (x,y)coordinate 306A for the first corner 704 using any other suitabletechnique.

Returning to FIG. 6 at step 604, the tracking system 100 determines(x,y) coordinates 306 for the markers 304 on the marker grid 702.Returning to the example in FIG. 7 , the tracking system 100 determinesa second (x,y) coordinate 306B for a first marker 304A on the markergrid 702. The tracking system 100 comprises marker grid information 716that identifies offsets between markers 304 on the marker grid 702 andthe first corner 704 of the marker grid 702. In this example, the offsetcomprises a distance between the first corner 704 of the marker grid 702and the first marker 304A with respect to the x-axis and the y-axis ofthe global plane 104. Using the marker grid information 1912, thetracking system 100 is able to determine the second (x,y) coordinate306B for the first marker 304A by adding an offset associated with thefirst marker 304A to the first (x,y) coordinate 306A for the firstcorner 704 of the marker grid 702.

In one embodiment, the tracking system 100 determines the second (x,y)coordinate 306B based at least in part on a rotation of the marker grid702. For example, the tracking system 100 may receive a fourth (x,y)coordinate 306D that identifies x-value and a y-value in the globalplane 104 for a second corner 706 of the marker grid 702. The trackingsystem 100 may obtain the fourth (x,y) coordinate 306D for the secondcorner 706 of the marker grid 702 using a process similar to the processdescribed in step 602. The tracking system 100 determines a rotationangle 712 between the first (x,y) coordinate 306A for the first corner704 of the marker grid 702 and the fourth (x,y) coordinate 306D for thesecond corner 706 of the marker grid 702. In this example, the rotationangle 712 is about the first corner 704 of the marker grid 702 withinthe global plane 104. The tracking system 100 then determines the second(x,y) coordinate 306B for the first marker 304A by applying atranslation by adding the offset associated with the first marker 304Ato the first (x,y) coordinate 306A for the first corner 704 of themarker grid 702 and applying a rotation using the rotation angle 712about the first (x,y) coordinate 306A for the first corner 704 of themarker grid 702. In other examples, the tracking system 100 maydetermine the second (x,y) coordinate 306B for the first marker 304Ausing any other suitable technique.

The tracking system 100 may repeat this process for one or moreadditional markers 304 on the marker grid 702. For example, the trackingsystem 100 determines a third (x,y) coordinate 306C for a second marker304B on the marker grid 702. Here, the tracking system 100 uses themarker grid information 716 to identify an offset associated with thesecond marker 304A. The tracking system 100 is able to determine thethird (x,y) coordinate 306C for the second marker 304B by adding theoffset associated with the second marker 304B to the first (x,y)coordinate 306A for the first corner 704 of the marker grid 702. Inanother embodiment, the tracking system 100 determines a third (x,y)coordinate 306C for a second marker 304B based at least in part on arotation of the marker grid 702 using a process similar to the processdescribed above for the first marker 304A.

Once the tracking system 100 knows the physical location of the markers304 within the space 102, the tracking system 100 then determines wherethe markers 304 are located with respect to the pixels in the frame 302of a sensor 108. At step 606, the tracking system 100 receives a frame302 from a sensor 108. The frame 302 is of the global plane 104 thatincludes at least a portion of the marker grid 702 in the space 102. Theframe 302 comprises one or more markers 304 of the marker grid 702. Theframe 302 is configured similar to the frame 302 described in FIGS. 2-4. For example, the frame 302 comprises a plurality of pixels that areeach associated with a pixel location 402 within the frame 302. Thepixel location 402 identifies a pixel row and a pixel column where apixel is located. In one embodiment, each pixel is associated with apixel value 404 that indicates a depth or distance measurement. Forexample, a pixel value 404 may correspond with a distance between thesensor 108 and a surface within the space 102.

At step 610, the tracking system 100 identifies markers 304 within theframe 302 of the sensor 108. The tracking system 100 may identifymarkers 304 within the frame 302 using a process similar to the processdescribed in step 206 of FIG. 2 . For example, the tracking system 100may use object detection to identify markers 304 within the frame 302.Referring to the example in FIG. 7 , each marker 304 is a unique shapeor symbol. In other examples, each marker 304 may have any other uniquefeatures (e.g. shape, pattern, color, text, etc.). In this example, thetracking system 100 may search for objects within the frame 302 thatcorrespond with the known features of a marker 304. Tracking system 100may identify the first marker 304A, the second marker 304B, and anyother markers 304 on the marker grid 702.

In one embodiment, the tracking system 100 compares the features of theidentified markers 304 to the features of known markers 304 on themarker grid 702 using a marker dictionary 718. The marker dictionary 718identifies a plurality of markers 304 that are associated with a markergrid 702. In this example, the tracking system 100 may identify thefirst marker 304A by identifying a star on the marker grid 702,comparing the star to the symbols in the marker dictionary 718, anddetermining that the star matches one of the symbols in the markerdictionary 718 that corresponds with the first marker 304A. Similarly,the tracking system 100 may identify the second marker 304B byidentifying a triangle on the marker grid 702, comparing the triangle tothe symbols in the marker dictionary 718, and determining that thetriangle matches one of the symbols in the marker dictionary 718 thatcorresponds with the second marker 304B. The tracking system 100 mayrepeat this process for any other identified markers 304 in the frame302.

In another embodiment, the marker grid 702 may comprise markers 304 thatcontain text. In this example, each marker 304 can be uniquelyidentified based on its text. This configuration allows the trackingsystem 100 to identify markers 304 in the frame 302 by using textrecognition or optical character recognition techniques on the frame302. In this case, the tracking system 100 may use a marker dictionary718 that comprises a plurality of predefined words that are eachassociated with a marker 304 on the marker grid 702. For example, thetracking system 100 may perform text recognition to identify text withthe frame 302. The tracking system 100 may then compare the identifiedtext to words in the marker dictionary 718. Here, the tracking system100 checks whether the identified text matched any of the known textthat corresponds with a marker 304 on the marker grid 702. The trackingsystem 100 may discard any text that does not match any words in themarker dictionary 718. When the tracking system 100 identifies text thatmatches a word in the marker dictionary 718, the tracking system 100 mayidentify the marker 304 that corresponds with the identified text. Forinstance, the tracking system 100 may determine that the identified textmatches the text associated with the first marker 304A. The trackingsystem 100 may identify the second marker 304B and any other markers 304on the marker grid 702 using a similar process.

Returning to FIG. 6 at step 610, the tracking system 100 determines anumber of identified markers 304 within the frame 302. Here, trackingsystem 100 counts the number of markers 304 that were detected withinthe frame 302. Referring to the example in FIG. 7 , the tracking system100 detects five markers 304 within the frame 302.

Returning to FIG. 6 at step 614, the tracking system 100 determineswhether the number of identified markers 304 is greater than or equal toa predetermined threshold value. The tracking system 100 may compare thenumber of identified markers 304 to the predetermined threshold valueusing a process similar to the process described in step 210 of FIG. 2 .The tracking system 100 returns to step 606 in response to determiningthat the number of identified markers 304 is less than the predeterminedthreshold value. In this case, the tracking system 100 returns to step606 to capture another frame 302 of the space 102 using the same sensor108 to try to detect more markers 304. Here, the tracking system 100tries to obtain a new frame 302 that includes a number of markers 304that is greater than or equal to the predetermined threshold value. Forexample, the tracking system 100 may receive new frame 302 of the space102 after an operator repositions the marker grid 702 within the space102. As another example, the tracking system 100 may receive new frame302 after lighting conditions have been changed to improve thedetectability of the markers 304 within the frame 302. In otherexamples, the tracking system 100 may receive new frame 302 after anykind of change that improves the detectability of the markers 304 withinthe frame 302.

The tracking system 100 proceeds to step 614 in response to determiningthat the number of identified markers 304 is greater than or equal tothe predetermined threshold value. Once the tracking system 100identifies a suitable number of markers 304 on the marker grid 702, thetracking system 100 then determines a pixel location 402 for each of theidentified markers 304. Each marker 304 may occupy multiple pixels inthe frame 302. This means that for each marker 304, the tracking system100 determines which pixel location 402 in the frame 302 correspondswith its (x,y) coordinate 306 in the global plane 104. In oneembodiment, the tracking system 100 using bounding boxes 708 to narrowor restrict the search space when trying to identify pixel location 402for markers 304. A bounding box 708 is a defined area or region withinthe frame 302 that contains a marker 304. For example, a bounding box708 may be defined as a set of pixels or a range of pixels of the frame302 that comprise a marker 304.

At step 614, the tracking system 100 identifies bounding boxes 708 formarkers 304 within the frame 302. In one embodiment, the tracking system100 identifies a plurality of pixels in the frame 302 that correspondwith a marker 304 and then defines a bounding box 708 that encloses thepixels corresponding with the marker 304. The tracking system 100 mayrepeat this process for each of the markers 304. Returning to theexample in FIG. 7 , the tracking system 100 may identify a firstbounding box 708A for the first marker 304A, a second bounding box 708Bfor the second marker 304B, and bounding boxes 708 for any otheridentified markers 304 within the frame 302.

In another embodiment, the tracking system may employ text or characterrecognition to identify the first marker 304A when the first marker 304Acomprises text. For example, the tracking system 100 may use textrecognition to identify pixels with the frame 302 that comprises a wordcorresponding with a marker 304. The tracking system 100 may then definea bounding box 708 that encloses the pixels corresponding with theidentified word. In other embodiments, the tracking system 100 mayemploy any other suitable image processing technique for identifyingbounding boxes 708 for the identified markers 304.

Returning to FIG. 6 at step 616, the tracking system 100 identifies apixel 710 within each bounding box 708 that corresponds with a pixellocation 402 in the frame 302 for a marker 304. As discussed above, eachmarker 304 may occupy multiple pixels in the frame 302 and the trackingsystem 100 determines which pixel 710 in the frame 302 corresponds withthe pixel location 402 for an (x,y) coordinate 306 in the global plane104. In one embodiment, each marker 304 comprises a light source.Examples of light sources include, but are not limited to, lightemitting diodes (LEDs), infrared (IR) LEDs, incandescent lights, or anyother suitable type of light source. In this configuration, a pixel 710corresponds with a light source for a marker 304. In another embodiment,each marker 304 may comprise a detectable feature that is unique to eachmarker 304. For example, each marker 304 may comprise a unique colorthat is associated with the marker 304. As another example, each marker304 may comprise a unique symbol or pattern that is associated with themarker 304. In this configuration, a pixel 710 corresponds with thedetectable feature for the marker 304. Continuing with the previousexample, the tracking system 100 identifies a first pixel 710A for thefirst marker 304, a second pixel 710B for the second marker 304, andpixels 710 for any other identified markers 304.

At step 618, the tracking system 100 determines pixel locations 402within the frame 302 for each of the identified pixels 710. For example,the tracking system 100 may identify a first pixel row and a first pixelcolumn of the frame 302 that corresponds with the first pixel 710A.Similarly, the tracking system 100 may identify a pixel row and a pixelcolumn in the frame 302 for each of the identified pixels 710.

The tracking system 100 generates a homography 118 for the sensor 108after the tracking system 100 determines (x,y) coordinates 306 in theglobal plane 104 and pixel locations 402 in the frame 302 for each ofthe identified markers 304. At step 620, the tracking system 100generates a homography 118 for the sensor 108 based on the pixellocations 402 of identified markers 304 in the frame 302 of the sensor108 and the (x,y) coordinate 306 of the identified markers 304 in theglobal plane 104. In one embodiment, the tracking system 100 correlatesthe pixel location 402 for each of the identified markers 304 with itscorresponding (x,y) coordinate 306. Continuing with the example in FIG.7 , the tracking system 100 associates the first pixel location 402 forthe first marker 304A with the second (x,y) coordinate 306B for thefirst marker 304A. The tracking system 100 also associates the secondpixel location 402 for the second marker 304B with the third (x,y)location 306C for the second marker 304B. The tracking system 100 mayrepeat this process for all of the identified markers 304.

The tracking system 100 then determines a relationship between the pixellocations 402 of identified markers 304 with the frame 302 of the sensor108 and the (x,y) coordinate 306 of the identified markers 304 in theglobal plane 104 to generate a homography 118 for the sensor 108. Thegenerated homography 118 allows the tracking system 100 to map pixellocations 402 in a frame 302 from the sensor 108 to (x,y) coordinates306 in the global plane 104. The generated homography 118 is similar tothe homography described in FIGS. 5A and 5B. Once the tracking system100 generates the homography 118 for the sensor 108, the tracking system100 stores an association between the sensor 108 and the generatedhomography 118 in memory (e.g. memory 3804).

The tracking system 100 may repeat the process described above togenerate and associate homographies 118 with other sensors 108. Themarker grid 702 may be moved or repositioned within the space 108 togenerate a homography 118 for another sensor 108. For example, anoperator may reposition the marker grid 702 to allow another sensor 108to view the markers 304 on the marker grid 702. As an example, thetracking system 100 may receive a second frame 302 from a second sensor108. In this example, the second frame 302 comprises the first marker304A and the second marker 304B. The tracking system 100 may determine athird pixel location 402 in the second frame 302 for the first marker304A and a fourth pixel location 402 in the second frame 302 for thesecond marker 304B. The tracking system 100 may then generate a secondhomography 118 based on the third pixel location 402 in the second frame302 for the first marker 304A, the fourth pixel location 402 in thesecond frame 302 for the second marker 304B, the (x,y) coordinate 306Bin the global plane 104 for the first marker 304A, the (x,y) coordinate306C in the global plane 104 for the second marker 304B, and pixellocations 402 and (x,y) coordinates 306 for other markers 304. Thesecond homography 118 comprises coefficients that translate betweenpixel locations 402 in the second frame 302 and physical locations (e.g.(x,y) coordinates 306) in the global plane 104. The coefficients of thesecond homography 118 are different from the coefficients of thehomography 118 that is associated with the first sensor 108. In otherwords, each sensor 108 is uniquely associated with a homography 118 thatmaps pixel locations 402 from the sensor 108 to physical locations inthe global plane 104. This process uniquely associates a homography 118to a sensor 108 based on the physical location (e.g. (x,y) coordinate306) of the sensor 108 in the global plane 104.

Shelf Position Calibration

FIG. 8 is a flowchart of an embodiment of a shelf position calibrationmethod 800 for the tracking system 100. The tracking system 100 mayemploy method 800 to periodically check whether a rack 112 or sensor 108has moved within the space 102. For example, a rack 112 may beaccidentally bumped or moved by a person which causes the rack's 112position to move with respect to the global plane 104. As anotherexample, a sensor 108 may come loose from its mounting structure whichcauses the sensor 108 to sag or move from its original location. Anychanges in the position of a rack 112 and/or a sensor 108 after thetracking system 100 has been calibrated will reduce the accuracy andperformance of the tracking system 100 when tracking objects within thespace 102. The tracking system 100 employs method 800 to detect wheneither a rack 112 or a sensor 108 has moved and then recalibrates itselfbased on the new position of the rack 112 or sensor 108.

A sensor 108 may be positioned within the space 102 such that frames 302captured by the sensor 108 will include one or more shelf markers 906that are located on a rack 112. A shelf marker 906 is an object that ispositioned on a rack 112 that can be used to determine a location (e.g.an (x,y) coordinate 306 and a pixel location 402) for the rack 112. Thetracking system 100 is configured to store the pixel locations 402 andthe (x,y) coordinates 306 of the shelf markers 906 that are associatedwith frames 302 from a sensor 108. In one embodiment, the pixellocations 402 and the (x,y) coordinates 306 of the shelf markers 906 maybe determined using a process similar to the process described in FIG. 2. In another embodiment, the pixel locations 402 and the (x,y)coordinates 306 of the shelf markers 906 may be provided by an operatoras an input to the tracking system 100.

A shelf marker 906 may be an object similar to the marker 304 describedin FIGS. 2-7 . In some embodiments, each shelf marker 906 on a rack 112is unique from other shelf markers 906 on the rack 112. This featureallows the tracking system 100 to determine an orientation of the rack112. Referring to the example in FIG. 9 , each shelf marker 906 is aunique shape that identifies a particular portion of the rack 112. Inthis example, the tracking system 100 may associate a first shelf marker906A and a second shelf marker 906B with a front of the rack 112.Similarly, the tracking system 100 may also associate a third shelfmarker 906C and a fourth shelf marker 906D with a back of the rack 112.In other examples, each shelf marker 906 may have any other uniquelyidentifiable features (e.g. color or patterns) that can be used toidentify a shelf marker 906.

Returning to FIG. 8 at step 802, the tracking system 100 receives afirst frame 302A from a first sensor 108. Referring to FIG. 9 as anexample, the first sensor 108 captures the first frame 302A whichcomprises at least a portion of a rack 112 within the global plane 104for the space 102.

Returning to FIG. 8 at step 804, the tracking system 100 identifies oneor more shelf markers 906 within the first frame 302A. Returning againto the example in FIG. 9 , the rack 112 comprises four shelf markers906. In one embodiment, the tracking system 100 may use object detectionto identify shelf markers 906 within the first frame 302A. For example,the tracking system 100 may search the first frame 302A for knownfeatures (e.g. shapes, patterns, colors, text, etc.) that correspondwith a shelf marker 906. In this example, the tracking system 100 mayidentify a shape (e.g. a star) in the first frame 302A that correspondswith a first shelf marker 906A. In other embodiments, the trackingsystem 100 may use any other suitable technique to identify a shelfmarker 906 within the first frame 302A. The tracking system 100 mayidentify any number of shelf markers 906 that are present in the firstframe 302A.

Once the tracking system 100 identifies one or more shelf markers 906that are present in the first frame 302A of the first sensor 108, thetracking system 100 then determines their pixel locations 402 in thefirst frame 302A so they can be compared to expected pixel locations 402for the shelf markers 906. Returning to FIG. 8 at step 806, the trackingsystem 100 determines current pixel locations 402 for the identifiedshelf markers 906 in the first frame 302A. Returning to the example inFIG. 9 , the tracking system 100 determines a first current pixellocation 402A for the shelf marker 906 within the first frame 302A. Thefirst current pixel location 402A comprises a first pixel row and firstpixel column where the shelf marker 906 is located within the firstframe 302A.

Returning to FIG. 8 at step 808, the tracking system 100 determineswhether the current pixel locations 402 for the shelf markers 906 matchthe expected pixel locations 402 for the shelf markers 906 in the firstframe 302A. Returning to the example in FIG. 9 , the tracking system 100determines whether the first current pixel location 402A matches a firstexpected pixel location 402 for the shelf marker 906. As discussedabove, when the tracking system 100 is initially calibrated, thetracking system 100 stores pixel location information 908 that comprisesexpected pixel locations 402 within the first frame 302A of the firstsensor 108 for shelf markers 906 of a rack 112. The tracking system 100uses the expected pixel locations 402 as reference points to determinewhether the rack 112 has moved. By comparing the expected pixel location402 for a shelf marker 906 with its current pixel location 402, thetracking system 100 can determine whether there are any discrepanciesthat would indicate that the rack 112 has moved.

The tracking system 100 may terminate method 800 in response todetermining that the current pixel locations 402 for the shelf markers906 in the first frame 302A match the expected pixel location 402 forthe shelf markers 906. In this case, the tracking system 100 determinesthat neither the rack 112 nor the first sensor 108 has moved since thecurrent pixel locations 402 match the expected pixel locations 402 forthe shelf marker 906.

The tracking system 100 proceeds to step 810 in response to adetermination at step 808 that one or more current pixel locations 402for the shelf markers 906 does not match an expected pixel location 402for the shelf markers 906. For example, the tracking system 100 maydetermine that the first current pixel location 402A does not match thefirst expected pixel location 402 for the shelf marker 906. In thiscase, the tracking system 100 determines that rack 112 and/or the firstsensor 108 has moved since the first current pixel location 402A doesnot match the first expected pixel location 402 for the shelf marker906. Here, the tracking system 100 proceeds to step 810 to identifywhether the rack 112 has moved or the first sensor 108 has moved.

At step 810, the tracking system 100 receives a second frame 302B from asecond sensor 108. The second sensor 108 is adjacent to the first sensor108 and has at least a partially overlapping field of view with thefirst sensor 108. The first sensor 108 and the second sensor 108 ispositioned such that one or more shelf markers 906 are observable byboth the first sensor 108 and the second sensor 108. In thisconfiguration, the tracking system 100 can use a combination ofinformation from the first sensor 108 and the second sensor 108 todetermine whether the rack 112 has moved or the first sensor 108 hasmoved. Returning to the example in FIG. 9 , the second frame 304Bcomprises the first shelf marker 906A, the second shelf marker 906B, thethird shelf marker 906C, and the fourth shelf marker 906D of the rack112.

Returning to FIG. 8 at step 812, the tracking system 100 identifies theshelf markers 906 that are present within the second frame 302B from thesecond sensor 108. The tracking system 100 may identify shelf markers906 using a process similar to the process described in step 804.Returning again to the example in FIG. 9 , tracking system 100 maysearch the second frame 302B for known features (e.g. shapes, patterns,colors, text, etc.) that correspond with a shelf marker 906. Forexample, the tracking system 100 may identify a shape (e.g. a star) inthe second frame 302B that corresponds with the first shelf marker 906A.

Once the tracking system 100 identifies one or more shelf markers 906that are present in the second frame 302B of the second sensor 108, thetracking system 100 then determines their pixel locations 402 in thesecond frame 302B so they can be compared to expected pixel locations402 for the shelf markers 906. Returning to FIG. 8 at step 814, thetracking system 100 determines current pixel locations 402 for theidentified shelf markers 906 in the second frame 302B. Returning to theexample in FIG. 9 , the tracking system 100 determines a second currentpixel location 402B for the shelf marker 906 within the second frame302B. The second current pixel location 402B comprises a second pixelrow and a second pixel column where the shelf marker 906 is locatedwithin the second frame 302B from the second sensor 108.

Returning to FIG. 8 at step 816, the tracking system 100 determineswhether the current pixel locations 402 for the shelf markers 906 matchthe expected pixel locations 402 for the shelf markers 906 in the secondframe 302B. Returning to the example in FIG. 9 , the tracking system 100determines whether the second current pixel location 402B matches asecond expected pixel location 402 for the shelf marker 906. Similar toas discussed above in step 808, the tracking system 100 stores pixellocation information 908 that comprises expected pixel locations 402within the second frame 302B of the second sensor 108 for shelf markers906 of a rack 112 when the tracking system 100 is initially calibrated.By comparing the second expected pixel location 402 for the shelf marker906 to its second current pixel location 402B, the tracking system 100can determine whether the rack 112 has moved or whether the first sensor108 has moved.

The tracking system 100 determines that the rack 112 has moved when thecurrent pixel location 402 and the expected pixel location 402 for oneor more shelf markers 906 do not match for multiple sensors 108. When arack 112 moves within the global plane 104, the physical location of theshelf markers 906 moves which causes the pixel locations 402 for theshelf markers 906 to also move with respect to any sensors 108 viewingthe shelf markers 906. This means that the tracking system 100 canconclude that the rack 112 has moved when multiple sensors 108 observe amismatch between current pixel locations 402 and expected pixellocations 402 for one or more shelf markers 906.

The tracking system 100 determines that the first sensor 108 has movedwhen the current pixel location 402 and the expected pixel location 402for one or more shelf markers 906 do not match only for the first sensor108. In this case, the first sensor 108 has moved with respect to therack 112 and its shelf markers 906 which causes the pixel locations 402for the shelf markers 906 to move with respect to the first sensor 108.The current pixel locations 402 of the shelf markers 906 will stillmatch the expected pixel locations 402 for the shelf markers 906 forother sensors 108 because the position of these sensors 108 and the rack112 has not changed.

The tracking system proceeds to step 818 in response to determining thatthe current pixel location 402 matches the second expected pixellocation 402 for the shelf marker 906 in the second frame 302B for thesecond sensor 108. In this case, the tracking system 100 determines thatthe first sensor 108 has moved. At step 818, the tracking system 100recalibrates the first sensor 108. In one embodiment, the trackingsystem 100 recalibrates the first sensor 108 by generating a newhomography 118 for the first sensor 108. The tracking system 100 maygenerate a new homography 118 for the first sensor 108 using shelfmarkers 906 and/or other markers 304. The tracking system 100 maygenerate the new homography 118 for the first sensor 108 using a processsimilar to the processes described in FIGS. 2 and/or 6 .

As an example, the tracking system 100 may use an existing homography118 that is currently associated with the first sensor 108 to determinephysical locations (e.g. (x,y) coordinates 306) for the shelf markers906. The tracking system 110 may then use the current pixel locations402 for the shelf markers 906 with their determined (x,y) coordinates306 to generate a new homography 118 for the first sensor 108. Forinstance, the tracking system 100 may use an existing homography 118that is associated with the first sensor 108 to determine a first (x,y)coordinate 306 in the global plane 104 where a first shelf marker 906 islocated, a second (x,y) coordinate 306 in the global plane 104 where asecond shelf marker 906 is located, and (x,y) coordinates 306 for anyother shelf markers 906. The tracking system 100 may apply the existinghomography 118 for the first sensor 108 to the current pixel location402 for the first shelf marker 906 in the first frame 302A to determinethe first (x,y) coordinate 306 for the first marker 906 using a processsimilar to the process described in FIG. 5A. The tracking system 100 mayrepeat this process for determining (x,y) coordinates 306 for any otheridentified shelf markers 906. Once the tracking system 100 determines(x,y) coordinates 306 for the shelf markers 906 and the current pixellocations 402 in the first frame 302A for the shelf markers 906, thetracking system 100 may then generate a new homography 118 for the firstsensor 108 using this information. For example, the tracking system 100may generate the new homography 118 based on the current pixel location402 for the first marker 906A, the current pixel location 402 for thesecond marker 906B, the first (x,y) coordinate 306 for the first marker906A, the second (x,y) coordinate 306 for the second marker 906B, and(x,y) coordinates 306 and pixel locations 402 for any other identifiedshelf markers 906 in the first frame 302A. The tracking system 100associates the first sensor 108 with the new homography 118. Thisprocess updates the homography 118 that is associated with the firstsensor 108 based on the current location of the first sensor 108.

In another embodiment, the tracking system 100 may recalibrate the firstsensor 108 by updating the stored expected pixel locations for the shelfmarker 906 for the first sensor 108. For example, the tracking system100 may replace the previous expected pixel location 402 for the shelfmarker 906 with its current pixel location 402. Updating the expectedpixel locations 402 for the shelf markers 906 with respect to the firstsensor 108 allows the tracking system 100 to continue to monitor thelocation of the rack 112 using the first sensor 108. In this case, thetracking system 100 can continue comparing the current pixel locations402 for the shelf markers 906 in the first frame 302A for the firstsensor 108 with the new expected pixel locations 402 in the first frame302A.

At step 820, the tracking system 100 sends a notification that indicatesthat the first sensor 108 has moved. Examples of notifications include,but are not limited to, text messages, short message service (SMS)messages, multimedia messaging service (MMS) messages, pushnotifications, application popup notifications, emails, or any othersuitable type of notifications. For example, the tracking system 100 maysend a notification indicating that the first sensor 108 has moved to aperson associated with the space 102. In response to receiving thenotification, the person may inspect and/or move the first sensor 108back to its original location.

Returning to step 816, the tracking system 100 proceeds to step 822 inresponse to determining that the current pixel location 402 does notmatch the expected pixel location 402 for the shelf marker 906 in thesecond frame 302B. In this case, the tracking system 100 determines thatthe rack 112 has moved. At step 822, the tracking system 100 updates theexpected pixel location information 402 for the first sensor 108 and thesecond sensor 108. For example, the tracking system 100 may replace theprevious expected pixel location 402 for the shelf marker 906 with itscurrent pixel location 402 for both the first sensor 108 and the secondsensor 108. Updating the expected pixel locations 402 for the shelfmarkers 906 with respect to the first sensor 108 and the second sensor108 allows the tracking system 100 to continue to monitor the locationof the rack 112 using the first sensor 108 and the second sensor 108. Inthis case, the tracking system 100 can continue comparing the currentpixel locations 402 for the shelf markers 906 for the first sensor 108and the second sensor 108 with the new expected pixel locations 402.

At step 824, the tracking system 100 sends a notification that indicatesthat the rack 112 has moved. For example, the tracking system 100 maysend a notification indicating that the rack 112 has moved to a personassociated with the space 102. In response to receiving thenotification, the person may inspect and/or move the rack 112 back toits original location. The tracking system 100 may update the expectedpixel locations 402 for the shelf markers 906 again once the rack 112 ismoved back to its original location.

Object Tracking Handoff

FIG. 10 is a flowchart of an embodiment of a tracking handoff method1000 for the tracking system 100. The tracking system 100 may employmethod 1000 to hand off tracking information for an object (e.g. aperson) as it moves between the fields of view of adjacent sensors 108.For example, the tracking system 100 may track the position of people(e.g. shoppers) as they move around within the interior of the space102. Each sensor 108 has a limited field of view which means that eachsensor 108 can only track the position of a person within a portion ofthe space 102. The tracking system 100 employs a plurality of sensors108 to track the movement of a person within the entire space 102. Eachsensor 108 operates independently from one another which means that thetracking system 100 keeps track of a person as they move from the fieldof view of one sensor 108 into the field of view of an adjacent sensor108.

The tracking system 100 is configured such that an object identifier1118 (e.g. a customer identifier) is assigned to each person as theyenter the space 102. The object identifier 1118 may be used to identifya person and other information associated with the person. Examples ofobject identifiers 1118 include, but are not limited to, names, customeridentifiers, alphanumeric codes, phone numbers, email addresses, or anyother suitable type of identifier for a person or object. In thisconfiguration, the tracking system 100 tracks a person's movement withinthe field of view of a first sensor 108 and then hands off trackinginformation (e.g. an object identifier 1118) for the person as it entersthe field of view of a second adjacent sensor 108.

In one embodiment, the tracking system 100 comprises adjacency lists1114 for each sensor 108 that identifies adjacent sensors 108 and thepixels within the frame 302 of the sensor 108 that overlap with theadjacent sensors 108. Referring to the example in FIG. 11 , a firstsensor 108 and a second sensor 108 have partially overlapping fields ofview. This means that a first frame 302A from the first sensor 108partially overlaps with a second frame 302B from the second sensor 108.The pixels that overlap between the first frame 302A and the secondframe 302B are referred to as an overlap region 1110. In this example,the tracking system 100 comprises a first adjacency list 1114A thatidentifies pixels in the first frame 302A that correspond with theoverlap region 1110 between the first sensor 108 and the second sensor108. For example, the first adjacency list 1114A may identify a range ofpixels in the first frame 302A that correspond with the overlap region1110. The first adjacency list 114A may further comprise informationabout other overlap regions between the first sensor 108 and otheradjacent sensors 108. For instance, a third sensor 108 may be configuredto capture a third frame 302 that partially overlaps with the firstframe 302A. In this case, the first adjacency list 1114A will furthercomprise information that identifies pixels in the first frame 302A thatcorrespond with an overlap region between the first sensor 108 and thethird sensor 108. Similarly, the tracking system 100 may furthercomprise a second adjacency list 1114B that is associated with thesecond sensor 108. The second adjacency list 1114B identifies pixels inthe second frame 302B that correspond with the overlap region 1110between the first sensor 108 and the second sensor 108. The secondadjacency list 1114B may further comprise information about otheroverlap regions between the second sensor 108 and other adjacent sensors108. In FIG. 11 , the second tracking list 1112B is shown as a separatedata structure from the first tracking list 1112A, however, the trackingsystem 100 may use a single data structure to store tracking listinformation that is associated with multiple sensors 108.

Once the first person 1106 enters the space 102, the tracking system 100will track the object identifier 1118 associated with the first person1106 as well as pixel locations 402 in the sensors 108 where the firstperson 1106 appears in a tracking list 1112. For example, the trackingsystem 100 may track the people within the field of view of a firstsensor 108 using a first tracking list 1112A, the people within thefield of view of a second sensor 108 using a second tracking list 1112B,and so on. In this example, the first tracking list 1112A comprisesobject identifiers 1118 for people being tracked using the first sensor108. The first tracking list 1112A further comprises pixel locationinformation that indicates the location of a person within the firstframe 302A of the first sensor 108. In some embodiments, the firsttracking list 1112A may further comprise any other suitable informationassociated with a person being tracked by the first sensor 108. Forexample, the first tracking list 1112A may identify (x,y) coordinates306 for the person in the global plane 104, previous pixel locations 402within the first frame 302A for a person, and/or a travel direction 1116for a person. For instance, the tracking system 100 may determine atravel direction 1116 for the first person 1106 based on their previouspixel locations 402 within the first frame 302A and may store thedetermined travel direction 1116 in the first tracking list 1112A. Inone embodiment, the travel direction 1116 may be represented as a vectorwith respect to the global plane 104. In other embodiments, the traveldirection 1116 may be represented using any other suitable format.

Returning to FIG. 10 at step 1002, the tracking system 100 receives afirst frame 302A from a first sensor 108. Referring to FIG. 11 as anexample, the first sensor 108 captures an image or frame 302A of aglobal plane 104 for at least a portion of the space 102. In thisexample, the first frame 1102 comprises a first object (e.g. a firstperson 1106) and a second object (e.g. a second person 1108). In thisexample, the first frame 302A captures the first person 1106 and thesecond person 1108 as they move within the space 102.

Returning to FIG. 10 at step 1004, the tracking system 100 determines afirst pixel location 402A in the first frame 302A for the first person1106. Here, the tracking system 100 determines the current location forthe first person 1106 within the first frame 302A from the first sensor108. Continuing with the example in FIG. 11 , the tracking system 100identifies the first person 1106 in the first frame 302A and determinesa first pixel location 402A that corresponds with the first person 1106.In a given frame 302, the first person 1106 is represented by acollection of pixels within the frame 302. Referring to the example inFIG. 11 , the first person 1106 is represented by a collection of pixelsthat show an overhead view of the first person 1106. The tracking system100 associates a pixel location 402 with the collection of pixelsrepresenting the first person 1106 to identify the current location ofthe first person 1106 within a frame 302. In one embodiment, the pixellocation 402 of the first person 1106 may correspond with the head ofthe first person 1106. In this example, the pixel location 402 of thefirst person 1106 may be located at about the center of the collectionof pixels that represent the first person 1106. As another example, thetracking system 100 may determine a bounding box 708 that encloses thecollection of pixels in the first frame 302A that represent the firstperson 1106. In this example, the pixel location 402 of the first person1106 may be located at about the center of the bounding box 708.

As another example, the tracking system 100 may use object detection orcontour detection to identify the first person 1106 within the firstframe 302A. In this example, the tracking system 100 may identify one ormore features for the first person 1106 when they enter the space 102.The tracking system 100 may later compare the features of a person inthe first frame 302A to the features associated with the first person1106 to determine if the person is the first person 1106. In otherexamples, the tracking system 100 may use any other suitable techniquesfor identifying the first person 1106 within the first frame 302A. Thefirst pixel location 402A comprises a first pixel row and a first pixelcolumn that corresponds with the current location of the first person1106 within the first frame 302A.

Returning to FIG. 10 at step 1006, the tracking system 100 determinesthe object is within the overlap region 1110 between the first sensor108 and the second sensor 108. Returning to the example in FIG. 11 , thetracking system 100 may compare the first pixel location 402A for thefirst person 1106 to the pixels identified in the first adjacency list1114A that correspond with the overlap region 1110 to determine whetherthe first person 1106 is within the overlap region 1110. The trackingsystem 100 may determine that the first object 1106 is within theoverlap region 1110 when the first pixel location 402A for the firstobject 1106 matches or is within a range of pixels identified in thefirst adjacency list 1114A that corresponds with the overlap region1110. For example, the tracking system 100 may compare the pixel columnof the pixel location 402A with a range of pixel columns associated withthe overlap region 1110 and the pixel row of the pixel location 402Awith a range of pixel rows associated with the overlap region 1110 todetermine whether the pixel location 402A is within the overlap region1110. In this example, the pixel location 402A for the first person 1106is within the overlap region 1110.

At step 1008, the tracking system 100 applies a first homography 118 tothe first pixel location 402A to determine a first (x,y) coordinate 306in the global plane 104 for the first person 1106. The first homography118 is configured to translate between pixel locations 402 in the firstframe 302A and (x,y) coordinates 306 in the global plane 104. The firsthomography 118 is configured similar to the homography 118 described inFIGS. 2-5B. As an example, the tracking system 100 may identify thefirst homography 118 that is associated with the first sensor 108 andmay use matrix multiplication between the first homography 118 and thefirst pixel location 402A to determine the first (x,y) coordinate 306 inthe global plane 104.

At step 1010, the tracking system 100 identifies an object identifier1118 for the first person 1106 from the first tracking list 1112Aassociated with the first sensor 108. For example, the tracking system100 may identify an object identifier 1118 that is associated with thefirst person 1106. At step 1012, the tracking system 100 stores theobject identifier 1118 for the first person 1106 in a second trackinglist 1112B associated with the second sensor 108. Continuing with theprevious example, the tracking system 100 may store the objectidentifier 1118 for the first person 1106 in the second tracking list1112B. Adding the object identifier 1118 for the first person 1106 tothe second tracking list 1112B indicates that the first person 1106 iswithin the field of view of the second sensor 108 and allows thetracking system 100 to begin tracking the first person 1106 using thesecond sensor 108.

Once the tracking system 100 determines that the first person 1106 hasentered the field of view of the second sensor 108, the tracking system100 then determines where the first person 1106 is located in the secondframe 302B of the second sensor 108 using a homography 118 that isassociated with the second sensor 108. This process identifies thelocation of the first person 1106 with respect to the second sensor 108so they can be tracked using the second sensor 108. At step 1014, thetracking system 100 applies a homography 118 that is associated with thesecond sensor 108 to the first (x,y) coordinate 306 to determine asecond pixel location 402B in the second frame 302B for the first person1106. The homography 118 is configured to translate between pixellocations 402 in the second frame 302B and (x,y) coordinates 306 in theglobal plane 104. The homography 118 is configured similarly to thehomography 118 described in FIGS. 2-5B. As an example, the trackingsystem 100 may identify the homography 118 that is associated with thesecond sensor 108 and may use matrix multiplication between the inverseof the homography 118 and the first (x,y) coordinate 306 to determinethe second pixel location 402B in the second frame 302B.

At step 1016, the tracking system 100 stores the second pixel location402B with the object identifier 1118 for the first person 1106 in thesecond tracking list 1112B. In some embodiments, the tracking system 100may store additional information associated with the first person 1106in the second tracking list 1112B. For example, the tracking system 100may be configured to store a travel direction 1116 or any other suitabletype of information associated with the first person 1106 in the secondtracking list 1112B. After storing the second pixel location 402B in thesecond tracking list 1112B, the tracking system 100 may begin trackingthe movement of the person within the field of view of the second sensor108.

The tracking system 100 will continue to track the movement of the firstperson 1106 to determine when they completely leave the field of view ofthe first sensor 108. At step 1018, the tracking system 100 receives anew frame 302 from the first sensor 108. For example, the trackingsystem 100 may periodically receive additional frames 302 from the firstsensor 108. For instance, the tracking system 100 may receive a newframe 302 from the first sensor 108 every millisecond, every second,every five second, or at any other suitable time interval.

At step 1020, the tracking system 100 determines whether the firstperson 1106 is present in the new frame 302. If the first person 1106 ispresent in the new frame 302, then this means that the first person 1106is still within the field of view of the first sensor 108 and thetracking system 100 should continue to track the movement of the firstperson 1106 using the first sensor 108. If the first person 1106 is notpresent in the new frame 302, then this means that the first person 1106has left the field of view of the first sensor 108 and the trackingsystem 100 no longer needs to track the movement of the first person1106 using the first sensor 108. The tracking system 100 may determinewhether the first person 1106 is present in the new frame 302 using aprocess similar to the process described in step 1004. The trackingsystem 100 returns to step 1018 to receive additional frames 302 fromthe first sensor 108 in response to determining that the first person1106 is present in the new frame 1102 from the first sensor 108.

The tracking system 100 proceeds to step 1022 in response to determiningthat the first person 1106 is not present in the new frame 302. In thiscase, the first person 1106 has left the field of view for the firstsensor 108 and no longer needs to be tracked using the first sensor 108.At step 1022, the tracking system 100 discards information associatedwith the first person 1106 from the first tracking list 1112A. Once thetracking system 100 determines that the first person has left the fieldof view of the first sensor 108, then the tracking system 100 can stoptracking the first person 1106 using the first sensor 108 and can freeup resources (e.g. memory resources) that were allocated to tracking thefirst person 1106. The tracking system 100 will continue to track themovement of the first person 1106 using the second sensor 108 until thefirst person 1106 leaves the field of view of the second sensor 108. Forexample, the first person 1106 may leave the space 102 or may transitionto the field of view of another sensor 108.

Shelf Interaction Detection

FIG. 12 is a flowchart of an embodiment of a shelf interaction detectionmethod 1200 for the tracking system 100. The tracking system 100 mayemploy method 1200 to determine where a person is interacting with ashelf of a rack 112. In addition to tracking where people are locatedwithin the space 102, the tracking system 100 also tracks which items1306 a person picks up from a rack 112. As a shopper picks up items 1306from a rack 112, the tracking system 100 identifies and tracks whichitems 1306 the shopper has picked up, so they can be automatically addedto a digital cart 1410 that is associated with the shopper. This processallows items 1306 to be added to the person's digital cart 1410 withouthaving the shopper scan or otherwise identify the item 1306 they pickedup. The digital cart 1410 comprises information about items 1306 theshopper has picked up for purchase. In one embodiment, the digital cart1410 comprises item identifiers and a quantity associated with each itemin the digital cart 1410. For example, when the shopper picks up acanned beverage, an item identifier for the beverage is added to theirdigital cart 1410. The digital cart 1410 will also indicate the numberof beverages that the shopper has picked up. Once the shopper leaves thespace 102, the shopper will be automatically charged for the items 1306in their digital cart 1410.

In FIG. 13 , a side view of a rack 112 is shown from the perspective ofa person standing in front of the rack 112. In this example, the rack112 may comprise a plurality of shelves 1302 for holding and displayingitems 1306. Each shelf 1302 may be partitioned into one or more zones1304 for holding different items 1306. In FIG. 13 , the rack 112comprises a first shelf 1302A at a first height and a second shelf 1302Bat a second height. Each shelf 1302 is partitioned into a first zone1304A and a second zone 1304B. The rack 112 may be configured to carry adifferent item 1306 (i.e. items 1306A, 1306B, 1306C, and 1036D) withineach zone 1304 on each shelf 1302. In this example, the rack 112 may beconfigured to carry up to four different types of items 1306. In otherexamples, the rack 112 may comprise any other suitable number of shelves1302 and/or zones 1304 for holding items 1306. The tracking system 100may employ method 1200 to identify which item 1306 a person picks upfrom a rack 112 based on where the person is interacting with the rack112.

Returning to FIG. 12 at step 1202, the tracking system 100 receives aframe 302 from a sensor 108. Referring to FIG. 14 as an example, thesensor 108 captures a frame 302 of at least a portion of the rack 112within the global plane 104 for the space 102. In FIG. 14 , an overheadview of the rack 112 and two people standing in front of the rack 112 isshown from the perspective of the sensor 108. The frame 302 comprises aplurality of pixels that are each associated with a pixel location 402for the sensor 108. Each pixel location 402 comprises a pixel row, apixel column, and a pixel value. The pixel row and the pixel columnindicate the location of a pixel within the frame 302 of the sensor 108.The pixel value corresponds with a z-coordinate (e.g. a height) in theglobal plane 104. The z-coordinate corresponds with a distance betweensensor 108 and a surface in the global plane 104.

The frame 302 further comprises one or more zones 1404 that areassociated with zones 1304 of the rack 112. Each zone 1404 in the frame302 corresponds with a portion of the rack 112 in the global plane 104.Referring to the example in FIG. 14 , the frame 302 comprises a firstzone 1404A and a second zone 1404B that are associated with the rack112. In this example, the first zone 1404A and the second zone 1404Bcorrespond with the first zone 1304A and the second zone 1304B of therack 112, respectively.

The frame 302 further comprises a predefined zone 1406 that is used as avirtual curtain to detect where a person 1408 is interacting with therack 112. The predefined zone 1406 is an invisible barrier defined bythe tracking system 100 that the person 1408 reaches through to pick upitems 1306 from the rack 112. The predefined zone 1406 is locatedproximate to the one or more zones 1304 of the rack 112. For example,the predefined zone 1406 may be located proximate to the front of theone or more zones 1304 of the rack 112 where the person 1408 would reachto grab for an item 1306 on the rack 112. In some embodiments, thepredefined zone 1406 may at least partially overlap with the first zone1404A and the second zone 1404B.

Returning to FIG. 12 at step 1204, the tracking system 100 identifies anobject within a predefined zone 1406 of the frame 1402. For example, thetracking system 100 may detect that the person's 1408 hand enters thepredefined zone 1406. In one embodiment, the tracking system 100 maycompare the frame 1402 to a previous frame that was captured by thesensor 108 to detect that the person's 1408 hand has entered thepredefined zone 1406. In this example, the tracking system 100 may usedifferences between the frames 302 to detect that the person's 1408 handenters the predefined zone 1406. In other embodiments, the trackingsystem 100 may employ any other suitable technique for detecting whenthe person's 1408 hand has entered the predefined zone 1406.

In one embodiment, the tracking system 100 identifies the rack 112 thatis proximate to the person 1408. Returning to the example in FIG. 14 ,the tracking system 100 may determine a pixel location 402A in the frame302 for the person 1408. The tracking system 100 may determine a pixellocation 402A for the person 1408 using a process similar to the processdescribed in step 1004 of FIG. 10 . The tracking system 100 may use ahomography 118 associated with the sensor 108 to determine an (x,y)coordinate 306 in the global plane 104 for the person 1408. Thehomography 118 is configured to translate between pixel locations 402 inthe frame 302 and (x,y) coordinates 306 in the global plane 104. Thehomography 118 is configured similarly to the homography 118 describedin FIGS. 2-5B. As an example, the tracking system 100 may identify thehomography 118 that is associated with the sensor 108 and may use matrixmultiplication between the homography 118 and the pixel location 402A ofthe person 1408 to determine an (x,y) coordinate 306 in the global plane104. The tracking system 100 may then identify which rack 112 is closestto the person 1408 based on the person's 1408 (x,y) coordinate 306 inthe global plane 104.

The tracking system 100 may identify an item map 1308 corresponding withthe rack 112 that is closest to the person 1408. In one embodiment, thetracking system 100 comprises an item map 1308 that associates items1306 with particular locations on the rack 112. For example, an item map1308 may comprise a rack identifier and a plurality of item identifiers.Each item identifier is mapped to a particular location on the rack 112.Returning to the example in FIG. 13 , a first item 1306A is mapped to afirst location that identifies the first zone 1304A and the first shelf1302A of the rack 112, a second item 1306B is mapped to a secondlocation that identifies the second zone 1304B and the first shelf 1302Aof the rack 112, a third item 1306C is mapped to a third location thatidentifies the first zone 1304A and the second shelf 1302B of the rack112, and a fourth item 1306D is mapped to a fourth location thatidentifies the second zone 1304B and the second shelf 1302B of the rack112.

Returning to FIG. 12 at step 1206, the tracking system 100 determines apixel location 402B in the frame 302 for the object that entered thepredefined zone 1406. Continuing with the previous example, the pixellocation 402B comprises a first pixel row, a first pixel column, and afirst pixel value for the person's 1408 hand. In this example, theperson's 1408 hand is represented by a collection of pixels in thepredefined zone 1406. In one embodiment, the pixel location 402 of theperson's 1408 hand may be located at about the center of the collectionof pixels that represent the person's 1408 hand. In other examples, thetracking system 100 may use any other suitable technique for identifyingthe person's 1408 hand within the frame 302.

Once the tracking system 100 determines the pixel location 402B of theperson's 1408 hand, the tracking system 100 then determines which shelf1302 and zone 1304 of the rack 112 the person 1408 is reaching for. Atstep 1208, the tracking system 100 determines whether the pixel location402B for the object (i.e. the person's 1408 hand) corresponds with afirst zone 1304A of the rack 112. The tracking system 100 uses the pixellocation 402B of the person's 1408 hand to determine which side of therack 112 the person 1408 is reaching into. Here, the tracking system 100checks whether the person is reaching for an item on the left side ofthe rack 112.

Each zone 1304 of the rack 112 is associated with a plurality of pixelsin the frame 302 that can be used to determine where the person 1408 isreaching based on the pixel location 402B of the person's 1408 hand.Continuing with the example in FIG. 14 , the first zone 1304A of therack 112 corresponds with the first zone 1404A which is associated witha first range of pixels 1412 in the frame 302. Similarly, the secondzone 1304B of the rack 112 corresponds with the second zone 1404B whichis associated with a second range of pixels 1414 in the frame 302. Thetracking system 100 may compare the pixel location 402B of the person's1408 hand to the first range of pixels 1412 to determine whether thepixel location 402B corresponds with the first zone 1304A of the rack112. In this example, the first range of pixels 1412 corresponds with arange of pixel columns in the frame 302. In other examples, the firstrange of pixels 1412 may correspond with a range of pixel rows or acombination of pixel row and columns in the frame 302.

In this example, the tracking system 100 compares the first pixel columnof the pixel location 402B to the first range of pixels 1412 todetermine whether the pixel location 1410 corresponds with the firstzone 1304A of the rack 112. In other words, the tracking system 100compares the first pixel column of the pixel location 402B to the firstrange of pixels 1412 to determine whether the person 1408 is reachingfor an item 1306 on the left side of the rack 112. In FIG. 14 , thepixel location 402B for the person's 1408 hand does not correspond withthe first zone 1304A of the rack 112. The tracking system 100 proceedsto step 1210 in response to determining that the pixel location 402B forthe object corresponds with the first zone 1304A of the rack 112. Atstep 1210, the tracking system 100 identifies the first zone 1304A ofthe rack 112 based on the pixel location 402B for the object thatentered the predefined zone 1406. In this case, the tracking system 100determines that the person 1408 is reaching for an item on the left sideof the rack 112.

Returning to step 1208, the tracking system 100 proceeds to step 1212 inresponse to determining that the pixel location 402B for the object thatentered the predefined zone 1406 does not correspond with the first zone1304B of the rack 112. At step 1212, the tracking system 100 identifiesthe second zone 1304B of the rack 112 based on the pixel location 402Bof the object that entered the predefined zone 1406. In this case, thetracking system 100 determines that the person 1408 is reaching for anitem on the right side of the rack 112.

In other embodiments, the tracking system 100 may compare the pixellocation 402B to other ranges of pixels that are associated with otherzones 1304 of the rack 112. For example, the tracking system 100 maycompare the first pixel column of the pixel location 402B to the secondrange of pixels 1414 to determine whether the pixel location 402Bcorresponds with the second zone 1304B of the rack 112. In other words,the tracking system 100 compares the first pixel column of the pixellocation 402B to the second range of pixels 1414 to determine whetherthe person 1408 is reaching for an item 1306 on the right side of therack 112.

Once the tracking system 100 determines which zone 1304 of the rack 112the person 1408 is reaching into, the tracking system 100 thendetermines which shelf 1302 of the rack 112 the person 1408 is reachinginto. At step 1214, the tracking system 100 identifies a pixel value atthe pixel location 402B for the object that entered the predefined zone1406. The pixel value is a numeric value that corresponds with az-coordinate or height in the global plane 104 that can be used toidentify which shelf 1302 the person 1408 was interacting with. Thepixel value can be used to determine the height the person's 1408 handwas at when it entered the predefined zone 1406 which can be used todetermine which shelf 1302 the person 1408 was reaching into.

At step 1216, the tracking system 100 determines whether the pixel valuecorresponds with the first shelf 1302A of the rack 112. Returning to theexample in FIG. 13 , the first shelf 1302A of the rack 112 correspondswith a first range of z-values or heights 1310A and the second shelf1302B corresponds with a second range of z-values or heights 1310B. Thetracking system 100 may compare the pixel value to the first range ofz-values 1310A to determine whether the pixel value corresponds with thefirst shelf 1302A of the rack 112. As an example, the first range ofz-values 1310A may be a range between 2 meters and 1 meter with respectto the z-axis in the global plane 104. The second range of z-values1310B may be a range between 0.9 meters and 0 meters with respect to thez-axis in the global plane 104. The pixel value may have a value thatcorresponds with 1.5 meters with respect to the z-axis in the globalplane 104. In this example, the pixel value is within the first range ofz-values 1310A which indicates that the pixel value corresponds with thefirst shelf 1302A of the rack 112. In other words, the person's 1408hand was detected at a height that indicates the person 1408 wasreaching for the first shelf 1302A of the rack 112. The tracking system100 proceeds to step 1218 in response to determining that the pixelvalue corresponds with the first shelf of the rack 112. At step 1218,the tracking system 100 identifies the first shelf 1302A of the rack 112based on the pixel value.

Returning to step 1216, the tracking system 100 proceeds to step 1220 inresponse to determining that the pixel value does not correspond withthe first shelf 1302A of the rack 112. At step 1220, the tracking system100 identifies the second shelf 1302B of the rack 112 based on the pixelvalue. In other embodiments, the tracking system 100 may compare thepixel value to other z-value ranges that are associated with othershelves 1302 of the rack 112. For example, the tracking system 100 maycompare the pixel value to the second range of z-values 1310B todetermine whether the pixel value corresponds with the second shelf1302B of the rack 112.

Once the tracking system 100 determines which side of the rack 112 andwhich shelf 1302 of the rack 112 the person 1408 is reaching into, thenthe tracking system 100 can identify an item 1306 that corresponds withthe identified location on the rack 112. At step 1222, the trackingsystem 100 identifies an item 1306 based on the identified zone 1304 andthe identified shelf 1302 of the rack 112. The tracking system 100 usesthe identified zone 1304 and the identified shelf 1302 to identify acorresponding item 1306 in the item map 1308. Returning to the examplein FIG. 14 , the tracking system 100 may determine that the person 1408is reaching into the right side (i.e. zone 1404B) of the rack 112 andthe first shelf 1302A of the rack 112. In this example, the trackingsystem 100 determines that the person 1408 is reaching for and picked upitem 1306B from the rack 112.

In some instances, multiple people may be near the rack 112 and thetracking system 100 may need to determine which person is interactingwith the rack 112 so that it can add a picked-up item 1306 to theappropriate person's digital cart 1410. Returning to the example in FIG.14 , a second person 1420 is also near the rack 112 when the firstperson 1408 is picking up an item 1306 from the rack 112. In this case,the tracking system 100 should assign any picked-up items to the firstperson 1408 and not the second person 1420.

In one embodiment, the tracking system 100 determines which personpicked up an item 1306 based on their proximity to the item 1306 thatwas picked up. For example, the tracking system 100 may determine apixel location 402A in the frame 302 for the first person 1408. Thetracking system 100 may also identify a second pixel location 402C forthe second person 1420 in the frame 302. The tracking system 100 maythen determine a first distance 1416 between the pixel location 402A ofthe first person 1408 and the location on the rack 112 where the item1306 was picked up. The tracking system 100 also determines a seconddistance 1418 between the pixel location 402C of the second person 1420and the location on the rack 112 where the item 1306 was picked up. Thetracking system 100 may then determine that the first person 1408 iscloser to the item 1306 than the second person 1420 when the firstdistance 1416 is less than the second distance 1418. In this example,the tracking system 100 identifies the first person 1408 as the personthat most likely picked up the item 1306 based on their proximity to thelocation on the rack 112 where the item 1306 was picked up. This processallows the tracking system 100 to identify the correct person thatpicked up the item 1306 from the rack 112 before adding the item 1306 totheir digital cart 1410.

Returning to FIG. 12 at step 1224, the tracking system 100 adds theidentified item 1306 to a digital cart 1410 associated with the person1408. In one embodiment, the tracking system 100 uses weight sensors 110to determine a number of items 1306 that were removed from the rack 112.For example, the tracking system 100 may determine a weight decreaseamount on a weight sensor 110 after the person 1408 removes one or moreitems 1306 from the weight sensor 110. The tracking system 100 may thendetermine an item quantity based on the weight decrease amount. Forexample, the tracking system 100 may determine an individual item weightfor the items 1306 that are associated with the weight sensor 110. Forinstance, the weight sensor 110 may be associated with an item 1306 thatthat has an individual weight of sixteen ounces. When the weight sensor110 detects a weight decrease of sixty-four ounces, the weight sensor110 may determine that four of the items 1306 were removed from theweight sensor 110. In other embodiments, the digital cart 1410 mayfurther comprise any other suitable type of information associated withthe person 1408 and/or items 1306 that they have picked up.

Item Assignment Using a Local Zone

FIG. 15 is a flowchart of an embodiment of an item assigning method 1500for the tracking system 100. The tracking system 100 may employ method1500 to detect when an item 1306 has been picked up from a rack 112 andto determine which person to assign the item to using a predefined zone1808 that is associated with the rack 112. In a busy environment, suchas a store, there may be multiple people standing near a rack 112 whenan item is removed from the rack 112. Identifying the correct personthat picked up the item 1306 can be challenging. In this case, thetracking system 100 uses a predefined zone 1808 that can be used toreduce the search space when identifying a person that picks up an item1306 from a rack 112. The predefined zone 1808 is associated with therack 112 and is used to identify an area where a person can pick up anitem 1306 from the rack 112. The predefined zone 1808 allows thetracking system 100 to quickly ignore people are not within an areawhere a person can pick up an item 1306 from the rack 112, for examplebehind the rack 112. Once the item 1306 and the person have beenidentified, the tracking system 100 will add the item to a digital cart1410 that is associated with the identified person.

At step 1502, the tracking system 100 detects a weight decrease on aweight sensor 110. Referring to FIG. 18 as an example, the weight sensor110 is disposed on a rack 112 and is configured to measure a weight forthe items 1306 that are placed on the weight sensor 110. In thisexample, the weight sensor 110 is associated with a particular item1306. The tracking system 100 detects a weight decrease on the weightsensor 110 when a person 1802 removes one or more items 1306 from theweight sensor 110.

Returning to FIG. 15 at step 1504, the tracking system 100 identifies anitem 1306 associated with the weight sensor 110. In one embodiment, thetracking system 100 comprises an item map 1308A that associates items1306 with particular locations (e.g. zones 1304 and/or shelves 1302) andweight sensors 110 on the rack 112. For example, an item map 1308A maycomprise a rack identifier, weight sensor identifiers, and a pluralityof item identifiers. Each item identifier is mapped to a particularweight sensor 110 (i.e. weight sensor identifier) on the rack 112. Thetracking system 100 determines which weight sensor 110 detected a weightdecrease and then identifies the item 1306 or item identifier thatcorresponds with the weight sensor 110 using the item map 1308A.

At step 1506, the tracking system 100 receives a frame 302 of the rack112 from a sensor 108. The sensor 108 captures a frame 302 of at least aportion of the rack 112 within the global plane 104 for the space 102.The frame 302 comprises a plurality of pixels that are each associatedwith a pixel location 402. Each pixel location 402 comprises a pixel rowand a pixel column. The pixel row and the pixel column indicate thelocation of a pixel within the frame 302.

The frame 302 comprises a predefined zone 1808 that is associated withthe rack 112. The predefined zone 1808 is used for identifying peoplethat are proximate to the front of the rack 112 and in a suitableposition for retrieving items 1306 from the rack 112. For example, therack 112 comprises a front portion 1810, a first side portion 1812, asecond side portion 1814, and a back portion 1814. In this example, aperson may be able to retrieve items 1306 from the rack 112 when theyare either in front or to the side of the rack 112. A person is unableto retrieve items 1306 from the rack 112 when they are behind the rack112. In this case, the predefined zone 1808 may overlap with at least aportion of the front portion 1810, the first side portion 1812, and thesecond side portion 1814 of the rack 112 in the frame 1806. Thisconfiguration prevents people that are behind the rack 112 from beingconsidered as a person who picked up an item 1306 from the rack 112. InFIG. 18 , the predefined zone 1808 is rectangular. In other examples,the predefined zone 1808 may be semi-circular or in any other suitableshape.

After the tracking system 100 determines that an item 1306 has beenpicked up from the rack 112, the tracking system 100 then begins toidentify people within the frame 302 that may have picked up the item1306 from the rack 112. At step 1508, the tracking system 100 identifiesa person 1802 within the frame 302. The tracking system 100 may identifya person 1802 within the frame 302 using a process similar to theprocess described in step 1004 of FIG. 10 . In other examples, thetracking system 100 may employ any other suitable technique foridentifying a person 1802 within the frame 302.

At step 1510, the tracking system 100 determines a pixel location 402Ain the frame 302 for the identified person 1802. The tracking system 100may determine a pixel location 402A for the identified person 1802 usinga process similar to the process described in step 1004 of FIG. 10 . Thepixel location 402A comprises a pixel row and a pixel column thatidentifies the location of the person 1802 in the frame 302 of thesensor 108.

At step 1511, the tracking system 100 applies a homography 118 to thepixel location 402A of the identified person 1802 to determine an (x,y)coordinate 306 in the global plane 104 for the identified person 1802.The homography 118 is configured to translate between pixel locations402 in the frame 302 and (x,y) coordinates 306 in the global plane 104.The homography 118 is configured similarly to the homography 118described in FIGS. 2-5B. As an example, the tracking system 100 mayidentify the homography 118 that is associated with the sensor 108 andmay use matrix multiplication between the homography 118 and the pixellocation 402A of the identified person 1802 to determine the (x,y)coordinate 306 in the global plane 104.

At step 1512, the tracking system 100 determines whether the identifiedperson 1802 is within a predefined zone 1808 associated with the rack112 in the frame 302. Continuing with the example in FIG. 18 , thepredefined zone 1808 is associated with a range of (x,y) coordinates 306in the global plane 104. The tracking system 100 may compare the (x,y)coordinate 306 for the identified person 1802 to the range of (x,y)coordinates 306 that are associated with the predefined zone 1808 todetermine whether the (x,y) coordinate 306 for the identified person1802 is within the predefined zone 1808. In other words, the trackingsystem 100 uses the (x,y) coordinate 306 for the identified person 1802to determine whether the identified person 1802 is within an areasuitable for picking up items 1306 from the rack 112. In this example,the (x,y) coordinate 306 for the person 1802 corresponds with a locationin front of the rack 112 and is within the predefined zone 1808 whichmeans that the identified person 1802 is in a suitable area forretrieving items 1306 from the rack 112.

In another embodiment, the predefined zone 1808 is associated with aplurality of pixels (e.g. a range of pixel rows and pixel columns) inthe frame 302. The tracking system 100 may compare the pixel location402A to the pixels associated with the predefined zone 1808 to determinewhether the pixel location 402A is within the predefined zone 1808. Inother words, the tracking system 100 uses the pixel location 402A of theidentified person 1802 to determine whether the identified person 1802is within an area suitable for picking up items 1306 from the rack 112.In this example, the tracking system 100 may compare the pixel column ofthe pixel location 402A with a range of pixel columns associated withthe predefined zone 1808 and the pixel row of the pixel location 402Awith a range of pixel rows associated with the predefined zone 1808 todetermine whether the identified person 1802 is within the predefinedzone 1808. In this example, the pixel location 402A for the person 1802is standing in front of the rack 112 and is within the predefined zone1808 which means that the identified person 1802 is in a suitable areafor retrieving items 1306 from the rack 112.

The tracking system 100 proceeds to step 1514 in response to determiningthat the identified person 1802 is within the predefined zone 1808.Otherwise, the tracking system 100 returns to step 1508 to identifyanother person within the frame 302. In this case, the tracking system100 determines the identified person 1802 is not in a suitable area forretrieving items 1306 from the rack 112, for example, the identifiedperson 1802 is standing behind the rack 112.

In some instances, multiple people may be near the rack 112 and thetracking system 100 may need to determine which person is interactingwith the rack 112 so that it can add a picked-up item 1306 to theappropriate person's digital cart 1410. Returning to the example in FIG.18 , a second person 1826 is standing next to the side of rack 112 inthe frame 302 when the first person 1802 picks up an item 1306 from therack 112. In this example, the second person 1826 is closer to the rack112 than the first person 1802, however, the tracking system 100 canignore the second person 1826 because the pixel location 402B of thesecond person 1826 is outside of the predetermined zone 1808 that isassociated with the rack 112. For example, the tracking system 100 mayidentify an (x,y) coordinate 306 in the global plane 104 for the secondperson 1826 and determine that the second person 1826 is outside of thepredefined zone 1808 based on their (x,y) coordinate 306. As anotherexample, the tracking system 100 may identify a pixel location 402Bwithin the frame 302 for the second person 1826 and determine that thesecond person 1826 is outside of the predefined zone 1808 based on theirpixel location 402B.

As another example, the frame 302 further comprises a third person 1832standing near the rack 112. In this case, the tracking system 100determines which person picked up the item 1306 based on their proximityto the item 1306 that was picked up. For example, the tracking system100 may determine an (x,y) coordinate 306 in the global plane 104 forthe third person 1832. The tracking system 100 may then determine afirst distance 1828 between the (x,y) coordinate 306 of the first person1802 and the location on the rack 112 where the item 1306 was picked up.The tracking system 100 also determines a second distance 1830 betweenthe (x,y) coordinate 306 of the third person 1832 and the location onthe rack 112 where the item 1306 was picked up. The tracking system 100may then determine that the first person 1802 is closer to the item 1306than the third person 1832 when the first distance 1828 is less than thesecond distance 1830. In this example, the tracking system 100identifies the first person 1802 as the person that most likely pickedup the item 1306 based on their proximity to the location on the rack112 where the item 1306 was picked up. This process allows the trackingsystem 100 to identify the correct person that picked up the item 1306from the rack 112 before adding the item 1306 to their digital cart1410.

As another example, the tracking system 100 may determine a pixellocation 402C in the frame 302 for a third person 1832. The trackingsystem 100 may then determine the first distance 1828 between the pixellocation 402A of the first person 1802 and the location on the rack 112where the item 1306 was picked up. The tracking system 100 alsodetermines the second distance 1830 between the pixel location 402C ofthe third person 1832 and the location on the rack 112 where the item1306 was picked up.

Returning to FIG. 15 at step 1514, the tracking system 100 adds the item1306 to a digital cart 1410 that is associated with the identifiedperson 1802. The tracking system 100 may add the item 1306 to thedigital cart 1410 using a process similar to the process described instep 1224 of FIG. 12 .

Item Identification

FIG. 16 is a flowchart of an embodiment of an item identification method1600 for the tracking system 100. The tracking system 100 may employmethod 1600 to identify an item 1306 that has a non-uniform weight andto assign the item 1306 to a person's digital cart 1410. For items 1306with a uniform weight, the tracking system 100 is able to determine thenumber of items 1306 that are removed from a weight sensor 110 based ona weight difference on the weight sensor 110. However, items 1306 suchas fresh food do not have a uniform weight which means that the trackingsystem 100 is unable to determine how many items 1306 were removed froma shelf 1302 based on weight measurements. In this configuration, thetracking system 100 uses a sensor 108 to identify markers 1820 (e.g.text or symbols) on an item 1306 that has been picked up and to identifya person near the rack 112 where the item 1306 was picked up. Forexample, a marker 1820 may be located on the packaging of an item 1806or on a strap for carrying the item 1806. Once the item 1306 and theperson have been identified, the tracking system 100 can add the item1306 to a digital cart 1410 that is associated with the identifiedperson.

At step 1602, the tracking system 100 detects a weight decrease on aweight sensor 110. Returning to the example in FIG. 18 , the weightsensor 110 is disposed on a rack 112 and is configured to measure aweight for the items 1306 that are placed on the weight sensor 110. Inthis example, the weight sensor 110 is associated with a particular item1306. The tracking system 100 detects a weight decrease on the weightsensor 110 when a person 1802 removes one or more items 1306 from theweight sensor 110.

After the tracking system 100 detects that an item 1306 was removed froma rack 112, the tracking system 100 will use a sensor 108 to identifythe item 1306 that was removed and the person who picked up the item1306. Returning to FIG. 16 at step 1604, the tracking system 100receives a frame 302 from a sensor 108. The sensor 108 captures a frame302 of at least a portion of the rack 112 within the global plane 104for the space 102. In the example shown in FIG. 18 , the sensor 108 isconfigured such that the frame 302 from the sensor 108 captures anoverhead view of the rack 112. The frame 302 comprises a plurality ofpixels that are each associated with a pixel location 402. Each pixellocation 402 comprises a pixel row and a pixel column. The pixel row andthe pixel column indicate the location of a pixel within the frame 302.

The frame 302 comprises a predefined zone 1808 that is configuredsimilar to the predefined zone 1808 described in step 1504 of FIG. 15 .In one embodiment, the frame 1806 may further comprise a secondpredefined zone that is configured as a virtual curtain similar to thepredefined zone 1406 that is described in FIGS. 12-14 . For example, thetracking system 100 may use the second predefined zone to detect thatthe person's 1802 hand reaches for an item 1306 before detecting theweight decrease on the weight sensor 110. In this example, the secondpredefined zone is used to alert the tracking system 100 that an item1306 is about to be picked up from the rack 112 which may be used totrigger the sensor 108 to capture a frame 302 that includes the item1306 being removed from the rack 112.

At step 1606, the tracking system 100 identifies a marker 1820 on anitem 1306 within a predefined zone 1808 in the frame 302. A marker 1820is an object with unique features that can be detected by a sensor 108.For instance, a marker 1820 may comprise a uniquely identifiable shape,color, symbol, pattern, text, a barcode, a QR code, or any othersuitable type of feature. The tracking system 100 may search the frame302 for known features that correspond with a marker 1820. Referring tothe example in FIG. 18 , the tracking system 100 may identify a shape(e.g. a star) on the packaging of the item 1806 in the frame 302 thatcorresponds with a marker 1820. As another example, the tracking system100 may use character or text recognition to identify alphanumeric textthat corresponds with a marker 1820 when the marker 1820 comprises text.In other examples, the tracking system 100 may use any other suitabletechnique to identify a marker 1820 within the frame 302.

Returning to FIG. 16 at step 1608, the tracking system 100 identifies anitem 1306 associated with the marker 1820. In one embodiment, thetracking system 100 comprises an item map 1308B that associates items1306 with particular markers 1820. For example, an item map 1308B maycomprise a plurality of item identifiers that are each mapped to aparticular marker 1820 (i.e. marker identifier). The tracking system 100identifies the item 1306 or item identifier that corresponds with themarker 1820 using the item map 1308B.

In some embodiments, the tracking system 100 may also use informationfrom a weight sensor 110 to identify the item 1306. For example, thetracking system 100 may comprise an item map 1308A that associates items1306 with particular locations (e.g. zone 1304 and/or shelves 1302) andweight sensors 110 on the rack 112. For example, an item map 1308A maycomprise a rack identifier, weight sensor identifiers, and a pluralityof item identifiers. Each item identifier is mapped to a particularweight sensor 110 (i.e. weight sensor identifier) on the rack 112. Thetracking system 100 determines which weight sensor 110 detected a weightdecrease and then identifies the item 1306 or item identifier thatcorresponds with the weight sensor 110 using the item map 1308A.

After the tracking system 100 identifies the item 1306 that was pickedup from the rack 112, the tracking system 100 then determines whichperson picked up the item 1306 from the rack 112. At step 1610, thetracking system 100 identifies a person 1802 within the frame 302. Thetracking system 100 may identify a person 1802 within the frame 302using a process similar to the process described in step 1004 of FIG. 10. In other examples, the tracking system 100 may employ any othersuitable technique for identifying a person 1802 within the frame 302.

At step 1612, the tracking system 100 determines a pixel location 402Afor the identified person 1802. The tracking system 100 may determine apixel location 402A for the identified person 1802 using a processsimilar to the process described in step 1004 of FIG. 10 . The pixellocation 402A comprises a pixel row and a pixel column that identifiesthe location of the person 1802 in the frame 302 of the sensor 108.

At step 1613, the tracking system 100 applies a homography 118 to thepixel location 402A of the identified person 1802 to determine an (x,y)coordinate 306 in the global plane 104 for the identified person 1802.The tracking system 100 may determine the (x,y) coordinate 306 in theglobal plane 104 for the identified person 1802 using a process similarto the process described in step 1511 of FIG. 15 .

At step 1614, the tracking system 100 determines whether the identifiedperson 1802 is within the predefined zone 1808. Here, the trackingsystem 100 determines whether the identified person 1802 is in asuitable area for retrieving items 1306 from the rack 112. The trackingsystem 100 may determine whether the identified person 1802 is withinthe predefined zone 1808 using a process similar to the processdescribed in step 1512 of FIG. 15 . The tracking system 100 proceeds tostep 1616 in response to determining that the identified person 1802 iswithin the predefined zone 1808. In this case, the tracking system 100determines the identified person 1802 is in a suitable area forretrieving items 1306 from the rack 112, for example the identifiedperson 1802 is standing in front of the rack 112. Otherwise, thetracking system 100 returns to step 1610 to identify another personwithin the frame 302. In this case, the tracking system 100 determinesthe identified person 1802 is not in a suitable area for retrievingitems 1306 from the rack 112, for example the identified person 1802 isstanding behind of the rack 112.

In some instances, multiple people may be near the rack 112 and thetracking system 100 may need to determine which person is interactingwith the rack 112 so that it can add a picked-up item 1306 to theappropriate person's digital cart 1410. The tracking system 100 mayidentify which person picked up the item 1306 from the rack 112 using aprocess similar to the process described in step 1512 of FIG. 15 .

At step 1614, the tracking system 100 adds the item 1306 to a digitalcart 1410 that is associated with the person 1802. The tracking system100 may add the item 1306 to the digital cart 1410 using a processsimilar to the process described in step 1224 of FIG. 12 .

Misplaced Item Identification

FIG. 17 is a flowchart of an embodiment of a misplaced itemidentification method 1700 for the tracking system 100. The trackingsystem 100 may employ method 1700 to identify items 1306 that have beenmisplaced on a rack 112. While a person is shopping, the shopper maydecide to put down one or more items 1306 that they have previouslypicked up. In this case, the tracking system 100 should identify whichitems 1306 were put back on a rack 112 and which shopper put the items1306 back so that the tracking system 100 can remove the items 1306 fromtheir digital cart 1410. Identifying an item 1306 that was put back on arack 112 is challenging because the shopper may not put the item 1306back in its correct location. For example, the shopper may put back anitem 1306 in the wrong location on the rack 112 or on the wrong rack112. In either of these cases, the tracking system 100 has to correctlyidentify both the person and the item 1306 so that the shopper is notcharged for item 1306 when they leave the space 102. In thisconfiguration, the tracking system 100 uses a weight sensor 110 to firstdetermine that an item 1306 was not put back in its correct location.The tracking system 100 then uses a sensor 108 to identify the personthat put the item 1306 on the rack 112 and analyzes their digital cart1410 to determine which item 1306 they most likely put back based on theweights of the items 1306 in their digital cart 1410.

At step 1702, the tracking system 100 detects a weight increase on aweight sensor 110. Returning to the example in FIG. 18 , a first person1802 places one or more items 1306 back on a weight sensor 110 on therack 112. The weight sensor 110 is configured to measure a weight forthe items 1306 that are placed on the weight sensor 110. The trackingsystem 100 detects a weight increase on the weight sensor 110 when aperson 1802 adds one or more items 1306 to the weight sensor 110.

At step 1704, the tracking system 100 determines a weight increaseamount on the weight sensor 110 in response to detecting the weightincrease on the weight sensor 110. The weight increase amountcorresponds with a magnitude of the weight change detected by the weightsensor 110. Here, the tracking system 100 determines how much of aweight increase was experienced by the weight sensor 110 after one ormore items 1306 were placed on the weight sensor 110.

In one embodiment, the tracking system 100 determines that the item 1306placed on the weight sensor 110 is a misplaced item 1306 based on theweight increase amount. For example, the weight sensor 110 may beassociated with an item 1306 that has a known individual item weight.This means that the weight sensor 110 is only expected to experienceweight changes that are multiples of the known item weight. In thisconfiguration, the tracking system 100 may determine that the returneditem 1306 is a misplaced item 1306 when the weight increase amount doesnot match the individual item weight or multiples of the individual itemweight for the item 1306 associated with the weight sensor 110. As anexample, the weight sensor 110 may be associated with an item 1306 thathas an individual weight of ten ounces. If the weight sensor 110 detectsa weight increase of twenty-five ounces, the tracking system 100 candetermine that the item 1306 placed weight sensor 114 is not an item1306 that is associated with the weight sensor 110 because the weightincrease amount does not match the individual item weight or multiplesof the individual item weight for the item 1306 that is associated withthe weight sensor 110.

After the tracking system 100 detects that an item 1306 has been placedback on the rack 112, the tracking system 100 will use a sensor 108 toidentify the person that put the item 1306 back on the rack 112. At step1706, the tracking system 100 receives a frame 302 from a sensor 108.The sensor 108 captures a frame 302 of at least a portion of the rack112 within the global plane 104 for the space 102. In the example shownin FIG. 18 , the sensor 108 is configured such that the frame 302 fromthe sensor 108 captures an overhead view of the rack 112. The frame 302comprises a plurality of pixels that are each associated with a pixellocation 402. Each pixel location 402 comprises a pixel row and a pixelcolumn. The pixel row and the pixel column indicate the location of apixel within the frame 302. In some embodiments, the frame 302 furthercomprises a predefined zone 1808 that is configured similar to thepredefined zone 1808 described in step 1504 of FIG. 15 .

At step 1708, the tracking system 100 identifies a person 1802 withinthe frame 302. The tracking system 100 may identify a person 1802 withinthe frame 302 using a process similar to the process described in step1004 of FIG. 10 . In other examples, the tracking system 100 may employany other suitable technique for identifying a person 1802 within theframe 302.

At step 1710, the tracking system 100 determines a pixel location 402Ain the frame 302 for the identified person 1802. The tracking system 100may determine a pixel location 402A for the identified person 1802 usinga process similar to the process described in step 1004 of FIG. 10 . Thepixel location 402A comprises a pixel row and a pixel column thatidentifies the location of the person 1802 in the frame 302 of thesensor 108.

At step 1712, the tracking system 100 determines whether the identifiedperson 1802 is within a predefined zone 1808 of the frame 302. Here, thetracking system 100 determines whether the identified person 1802 is ina suitable area for putting items 1306 back on the rack 112. Thetracking system 100 may determine whether the identified person 1802 iswithin the predefined zone 1808 using a process similar to the processdescribed in step 1512 of FIG. 15 . The tracking system 100 proceeds tostep 1714 in response to determining that the identified person 1802 iswithin the predefined zone 1808. In this case, the tracking system 100determines the identified person 1802 is in a suitable area for puttingitems 1306 back on the rack 112, for example the identified person 1802is standing in front of the rack 112. Otherwise, the tracking system 100returns to step 1708 to identify another person within the frame 302. Inthis case, the tracking system 100 determines the identified person isnot in a suitable area for retrieving items 1306 from the rack 112, forexample the person is standing behind of the rack 112.

In some instances, multiple people may be near the rack 112 and thetracking system 100 may need to determine which person is interactingwith the rack 112 so that it can remove the returned item 1306 from theappropriate person's digital cart 1410. The tracking system 100 maydetermine which person put back the item 1306 on the rack 112 using aprocess similar to the process described in step 1512 of FIG. 15 .

After the tracking system 100 identifies which person put back the item1306 on the rack 112, the tracking system 100 then determines which item1306 from the identified person's digital cart 1410 has a weight thatclosest matches the item 1306 that was put back on the rack 112. At step1714, the tracking system 100 identifies a plurality of items 1306 in adigital cart 1410 that is associated with the person 1802. Here, thetracking system 100 identifies the digital cart 1410 that is associatedwith the identified person 1802. For example, the digital cart 1410 maybe linked with the identified person's 1802 object identifier 1118. Inone embodiment, the digital cart 1410 comprises item identifiers thatare each associated with an individual item weight. At step 1716, thetracking system 100 identifies an item weight for each of the items 1306in the digital cart 1410. In one embodiment, the tracking system 100 maycomprise a set of item weights stored in memory and may look up the itemweight for each item 1306 using the item identifiers that are associatedwith the item's 1306 in the digital cart 1410.

At step 1718, the tracking system 100 identifies an item 1306 from thedigital cart 1410 with an item weight that closest matches the weightincrease amount. For example, the tracking system 100 may compare theweight increase amount measured by the weight sensor 110 to the itemweights associated with each of the items 1306 in the digital cart 1410.The tracking system 100 may then identify which item 1306 correspondswith an item weight that closest matches the weight increase amount.

In some cases, the tracking system 100 is unable to identify an item1306 in the identified person's digital cart 1410 that a weight thatmatches the measured weight increase amount on the weight sensor 110. Inthis case, the tracking system 100 may determine a probability that anitem 1306 was put down for each of the items 1306 in the digital cart1410. The probability may be based on the individual item weight and theweight increase amount. For example, an item 1306 with an individualweight that is closer to the weight increase amount will be associatedwith a higher probability than an item 1306 with an individual weightthat is further away from the weight increase amount.

In some instances, the probabilities are a function of the distancebetween a person and the rack 112. In this case, the probabilitiesassociated with items 1306 in a person's digital cart 1410 depend on howclose the person is to the rack 112 where the item 1306 was put back.For example, the probabilities associated with the items 1306 in thedigital cart 1410 may be inversely proportional to the distance betweenthe person and the rack 112. In other words, the probabilitiesassociated with the items in a person's digital cart 1410 decay as theperson moves further away from the rack 112. The tracking system 100 mayidentify the item 1306 that has the highest probability of being theitem 1306 that was put down.

In some cases, the tracking system 100 may consider items 1306 that arein multiple people's digital carts 1410 when there are multiple peoplewithin the predefined zone 1808 that is associated with the rack 112.For example, the tracking system 100 may determine a second person iswithin the predefined zone 1808 that is associated with the rack 112. Inthis example, the tracking system 100 identifies items 1306 from eachperson's digital cart 1410 that may correspond with the item 1306 thatwas put back on the rack 112 and selects the item 1306 with an itemweight that closest matches the item 1306 that was put back on the rack112. For instance, the tracking system 100 identifies item weights foritems 1306 in a second digital cart 1410 that is associated with thesecond person. The tracking system 100 identifies an item 1306 from thesecond digital cart 1410 with an item weight that closest matches theweight increase amount. The tracking system 100 determines a firstweight difference between a first identified item 1306 from digital cart1410 of the first person 1802 and the weight increase amount and asecond weight difference between a second identified item 1306 from thesecond digital cart 1410 of the second person. In this example, thetracking system 100 may determine that the first weight difference isless than the second weight difference, which indicates that the item1306 identified in the first person's digital cart 1410 closest matchesthe weight increase amount, and then removes the first identified item1306 from their digital cart 1410.

After the tracking system 100 identifies the item 1306 that most likelyput back on the rack 112 and the person that put the item 1306 back, thetracking system 100 removes the item 1306 from their digital cart 1410.At step 1720, the tracking system 100 removes the identified item 1306from the identified person's digital cart 1410. Here, the trackingsystem 100 discards information associated with the identified item 1306from the digital cart 1410. This process ensures that the shopper willnot be charged for item 1306 that they put back on a rack 112 regardlessof whether they put the item 1306 back in its correct location.

Auto-Exclusion Zones

In order to track the movement of people in the space 102, the trackingsystem 100 should generally be able to distinguish between the people(i.e., the target objects) and other objects (i.e., non-target objects),such as the racks 112, displays, and any other non-human objects in thespace 102. Otherwise, the tracking system 100 may waste memory andprocessing resources detecting and attempting to track these non-targetobjects. As described elsewhere in this disclosure (e.g., in FIGS. 24-26and corresponding description below), in some cases, people may betracked may be performed by detecting one or more contours in a set ofimage frames (e.g., a video) and monitoring movements of the contourbetween frames. A contour is generally a curve associated with an edgeof a representation of a person in an image. While the tracking system100 may detect contours in order to track people, in some instances, itmay be difficult to distinguish between contours that correspond topeople (e.g., or other target objects) and contours associated withnon-target objects, such as racks 112, signs, product displays, and thelike.

Even if sensors 108 are calibrated at installation to account for thepresence of non-target objects, in many cases, it may be challenging toreliably and efficiently recalibrate the sensors 108 to account forchanges in positions of non-target objects that should not be tracked inthe space 102. For example, if a rack 112, sign, product display, orother furniture or object in space 102 is added, removed, or moved(e.g., all activities which may occur frequently and which may occurwithout warning and/or unintentionally), one or more of the sensors 108may require recalibration or adjustment. Without this recalibration oradjustment, it is difficult or impossible to reliably track people inthe space 102. Prior to this disclosure, there was a lack of tools forefficiently recalibrating and/or adjusting sensors, such as sensors 108,in a manner that would provide reliable tracking.

This disclosure encompasses the recognition not only of the previouslyunrecognized problems described above (e.g., with respect to trackingpeople in space 102, which may change over time) but also providesunique solutions to these problems. As described in this disclosure,during an initial time period before people are tracked, pixel regionsfrom each sensor 108 may be determined that should be excluded duringsubsequent tracking. For example, during the initial time period, thespace 102 may not include any people such that contours detected by eachsensor 108 correspond only to non-target objects in the space for whichtracking is not desired. Thus, pixel regions, or “auto-exclusion zones,”corresponding to portions of each image generated by sensors 108 thatare not used for object detection and tracking (e.g., the pixelcoordinates of contours that should not be tracked). For instance, theauto-exclusion zones may correspond to contours detected in images thatare associated with non-target objects, contours that are spuriouslydetected at the edges of a sensor's field-of-view, and the like).Auto-exclusion zones can be determined automatically at any desired orappropriate time interval to improve the usability and performance ofthe tracking system 100.

After the auto-exclusion zones are determined, the tracking system 100may proceed to track people in the space 102. The auto-exclusion zonesare used to limit the pixel regions used by each sensor 108 for trackingpeople. For example, pixels corresponding to auto-exclusion zones may beignored by the tracking system 100 during tracking. In some cases, adetected person (or other target object) may be near or partiallyoverlapping with one or more auto-exclusion zones. In these cases, thetracking system 100 may determine, based on the extent to which apotential target object's position overlaps with the auto-exclusionzone, whether the target object will be tracked. This may reduce oreliminate false positive detection of non-target objects during persontracking in the space 102, while also improving the efficiency of thetracking system 100 by reducing wasted processing resources that wouldotherwise be expended attempting to track non-target objects. In someembodiments, a map of the space 102 may be generated that presents thephysical regions that are excluded during tracking (i.e., a map thatpresents a representation of the auto-exclusion zone(s) in the physicalcoordinates of the space). Such a map, for example, may facilitatetrouble-shooting of the tracking system by allowing an administrator tovisually confirm that people can be tracked in appropriate portions ofthe space 102.

FIG. 19 illustrates the determination of auto-exclusion zones 1910, 1914and the subsequent use of these auto-exclusion zones 1910, 1914 forimproved tracking of people (e.g., or other target objects) in the space102. In general, during an initial time period (t<t₀), top-view imageframes are received by the client(s) 105 and/or server 106 from sensors108 and used to determine auto-exclusion zones 1910, 1914. For instance,the initial time period at t<t₀ may correspond to a time when no peopleare in the space 102. For example, if the space 102 is open to thepublic during a portion of the day, the initial time period may bebefore the space 102 is opened to the public. In some embodiments, theserver 106 and/or client 105 may provide, for example, an alert ortransmit a signal indicating that the space 102 should be emptied ofpeople (e.g., or other target objects to be tracked) in order forauto-exclusion zones 1910, 1914 to be identified. In some embodiments, auser may input a command (e.g., via any appropriate interface coupled tothe server 106 and/or client(s) 105) to initiate the determination ofauto-exclusion zones 1910, 1914 immediately or at one or more desiredtimes in the future (e.g., based on a schedule).

An example top-view image frame 1902 used for determining auto-exclusionzones 1910, 1914 is shown in FIG. 19 . Image frame 1902 includes arepresentation of a first object 1904 (e.g., a rack 112) and arepresentation of a second object 1906. For instance, the first object1904 may be a rack 112, and the second object 1906 may be a productdisplay or any other non-target object in the space 102. In someembodiments, the second object 1906 may not correspond to an actualobject in the space but may instead be detected anomalously because oflighting in the space 102 and/or a sensor error. Each sensor 108generally generates at least one frame 1902 during the initial timeperiod, and these frame(s) 1902 is/are used to determine correspondingauto-exclusion zones 1910, 1914 for the sensor 108. For instance, thesensor client 105 may receive the top-view image 1902, and detectcontours (i.e., the dashed lines around zones 1910, 1914) correspondingto the auto-exclusion zones 1910, 1914 as illustrated in view 1908. Thecontours of auto-exclusion zones 1910, 1914 generally correspond tocurves that extend along a boundary (e.g., the edge) of objects 1904,1906 in image 1902. The view 1908 generally corresponds to apresentation of image 1902 in which the detected contours correspondingto auto-exclusion zones 1910, 1914 are presented but the correspondingobjects 1904, 1906, respectively, are not shown. For an image frame 1902that includes color and depth data, contours for auto-exclusion zones1910, 1914 may be determined at a given depth (e.g., a distance awayfrom sensor 108) based on the color data in the image 1902. For example,a steep gradient of a color value may correspond to an edge of an objectand is used to determine, or detect, a contour. For example, contoursfor the auto-exclusion zones 1910, 1914 may be determined using anysuitable contour or edge detection method such as Canny edge detection,threshold-based detection, or the like.

The client 105 determines pixel coordinates 1912 and 1916 correspondingto the locations of the auto-exclusions zones 1910 and 1914,respectively. The pixel coordinates 1912, 1916 generally correspond tothe locations (e.g., row and column numbers) in the image frame 1902that should be excluded during tracking. In general, objects associatedwith the pixel coordinates 1912, 1916 are not tracked by the trackingsystem 100. Moreover, certain objects which are detected outside of theauto-exclusion zones 1910, 1914 may not be tracked under certainconditions. For instance, if the position of the object (e.g., theposition associated with region 1920, discussed below with respect toview 1914) overlaps at least a threshold amount with an auto-exclusionzone 1910, 1914, the object may not be tracked. This prevents thetracking system 100 (i.e., or the local client 105 associated with asensor 108 or a subset of sensors 108) from attempting to unnecessarilytrack non-target objects. In some cases, auto-exclusion zones 1910, 1914correspond to non-target (e.g., inanimate) objects in the field-of-viewof a sensor 108 (e.g., a rack 112, which is associated with contour1910). However, auto-exclusion zones 1910, 1914 may also oralternatively correspond to other aberrant features or contours detectedby a sensor 108 (e.g., caused by sensor errors, inconsistent lighting,or the like).

Following the determination of pixel coordinates 1912, 1916 to excludeduring tracking, objects may be tracked during a subsequent time periodcorresponding to t>t₀. An example image frame 1918 generated duringtracking is shown in FIG. 19 . In frame 1918, region 1920 is detected aspossibly corresponding to what may or may not be a target object. Forexample, region 1920 may correspond to a pixel mask or bounding boxgenerated based on a contour detected in frame 1902. For example, apixel mask may be generated to fill in the area inside the contour or abounding box may be generated to encompass the contour. For example, apixel mask may include the pixel coordinates within the correspondingcontour. For instance, the pixel coordinates 1912 of auto-exclusion zone1910 may effectively correspond to a mask that overlays or “fills in”the auto-exclusion zone 1910. Following the detection of region 1920,the client 105 determines whether the region 1920 corresponds to atarget object which should tracked or is sufficiently overlapping withauto-exclusion zone 1914 to consider region 1920 as being associatedwith a non-target object. For example, the client 105 may determinewhether at least a threshold percentage of the pixel coordinates 1916overlap with (e.g., are the same as) pixel coordinates of region 1920.The overlapping region 1922 of these pixel coordinates is illustrated inframe 1918. For example, the threshold percentage may be about 50% ormore. In some embodiments, the threshold percentage may be as small asabout 10%. In response to determining that at least the thresholdpercentage of pixel coordinates overlap, the client 105 generally doesnot determine a pixel position for tracking the object associated withregion 1920. However, if overlap 1922 correspond to less than thethreshold percentage, an object associated with region 1920 is tracked,as described further below (e.g., with respect to FIGS. 24-26 ).

As described above, sensors 108 may be arranged such that adjacentsensors 108 have overlapping fields-of-view. For instance,fields-of-view of adjacent sensors 108 may overlap between about 10% to30%. As such, the same object may be detected by two different sensors108 and either included or excluded from tracking in the image framesreceived from each sensor 108 based on the unique auto-exclusion zonesdetermined for each sensor 108. This may facilitate more reliabletracking than was previously possible, even when one sensor 108 may havea large auto-exclusion zone (i.e., where a large proportion of pixelcoordinates in image frames generated by the sensor 108 are excludedfrom tracking). Accordingly, if one sensor 108 malfunctions, adjacentsensors 108 may still provide adequate tracking in the space 102.

If region 1920 corresponds to a target object (i.e., a person to trackin the space 102), the tracking system 100 proceeds to track the region1920. Example methods of tracking are described in greater detail belowwith respect to FIGS. 24-26 . In some embodiments, the server 106 usesthe pixel coordinates 1912, 1916 to determine corresponding physicalcoordinates (e.g., coordinates 2012, 2016 illustrated in FIG. 20 ,described below). For instance, the client 105 may determine pixelcoordinates 1912, 1916 corresponding to the local auto-exclusion zones1910, 1914 of a sensor 108 and transmit these coordinates 1912, 1916 tothe server 106. As shown in FIG. 20 , the server 106 may use the pixelcoordinates 1912, 1916 received from the sensor 108 to determinecorresponding physical coordinates 2010, 2016. For instance, ahomography generated for each sensor 108 (see FIGS. 2-7 and thecorresponding description above), which associates pixel coordinates(e.g., coordinates 1912, 1916) in an image generated by a given sensor108 to corresponding physical coordinates (e.g., coordinates 2012, 2016)in the space 102, may be employed to convert the excluded pixelcoordinates 1912, 1916 (of FIG. 19 ) to excluded physical coordinates2012, 2016 in the space 102. These excluded coordinates 2010, 2016 maybe used along with other coordinates from other sensors 108 to generatethe global auto-exclusion zone map 2000 of the space 102 which isillustrated in FIG. 20 . This map 2000, for example, may facilitatetrouble-shooting of the tracking system 100 by facilitatingquantification, identification, and/or verification of physical regions2002 of space 102 where objects may (and may not) be tracked. This mayallow an administrator or other individual to visually confirm thatobjects can be tracked in appropriate portions of the space 102). Ifregions 2002 correspond to known high-traffic zones of the space 102,system maintenance may be appropriate (e.g., which may involvereplacing, adjusting, and/or adding additional sensors 108).

FIG. 21 is a flowchart illustrating an example method 2100 forgenerating and using auto-exclusion zones (e.g., zones 1910, 1914 ofFIG. 19 ). Method 2100 may begin at step 2102 where one or more imageframes 1902 are received during an initial time period. As describedabove, the initial time period may correspond to an interval of timewhen no person is moving throughout the space 102, or when no person iswithin the field-of-view of one or more sensors 108 from which the imageframe(s) 1902 is/are received. In a typical embodiment, one or moreimage frames 1902 are generally received from each sensor 108 of thetracking system 100, such that local regions (e.g., auto-exclusion zones1910, 1914) to exclude for each sensor 108 may be determined. In someembodiments, a single image frame 1902 is received from each sensor 108to detect auto-exclusion zones 1910, 1914. However, in otherembodiments, multiple image frames 1902 are received from each sensor108. Using multiple image frames 1902 to identify auto-exclusions zones1910, 1914 for each sensor 108 may improve the detection of any spuriouscontours or other aberrations that correspond to pixel coordinates(e.g., coordinates 1912, 1916 of FIG. 19 ) which should be ignored orexcluded during tracking.

At step 2104, contours (e.g., dashed contour lines corresponding toauto-exclusion zones 1910, 1914 of FIG. 19 ) are detected in the one ormore image frames 1902 received at step 2102. Any appropriate contourdetection algorithm may be used including but not limited to those basedon Canny edge detection, threshold-based detection, and the like. Insome embodiments, the unique contour detection approaches described inthis disclosure may be used (e.g., to distinguish closely spacedcontours in the field-of-view, as described below, for example, withrespect to FIGS. 22 and 23 ). At step 2106, pixel coordinates (e.g.,coordinates 1912, 1916 of FIG. 19 ) are determined for the detectedcontours (from step 2104). The coordinates may be determined, forexample, based on a pixel mask that overlays the detected contours. Apixel mask may for example, correspond to pixels within the contours. Insome embodiments, pixel coordinates correspond to the pixel coordinateswithin a bounding box determined for the contour (e.g., as illustratedin FIG. 22 , described below). For instance, the bounding box may be arectangular box with an area that encompasses the detected contour. Atstep 2108, the pixel coordinates are stored. For instance, the client105 may store the pixel coordinates corresponding to auto-exclusionzones 1910, 1914 in memory (e.g., memory 3804 of FIG. 38 , describedbelow). As described above, the pixel coordinates may also oralternatively be transmitted to the server 106 (e.g., to generate a map2000 of the space, as illustrated in the example of FIG. 20 ).

At step 2110, the client 105 receives an image frame 1918 during asubsequent time during which tracking is performed (i.e., after thepixel coordinates corresponding to auto-exclusion zones are stored atstep 2108). The frame is received from sensor 108 and includes arepresentation of an object in the space 102. At step 2112, a contour isdetected in the frame received at step 2110. For example, the contourmay correspond to a curve along the edge of object represented in theframe 1902. The pixel coordinates determined at step 2106 may beexcluded (or not used) during contour detection. For instance, imagedata may be ignored and/or removed (e.g., given a value of zero, or thecolor equivalent) at the pixel coordinates determined at step 2106, suchthat no contours are detected at these coordinates. In some cases, acontour may be detected outside of these coordinates. In some cases, acontour may be detected that is partially outside of these coordinatesbut overlaps partially with the coordinates (e.g., as illustrated inimage 1918 of FIG. 19 ).

At step 2114, the client 105 generally determines whether the detectedcontour has a pixel position that sufficiently overlaps with pixelcoordinates of the auto-exclusion zones 1910, 1914 determined at step2106. If the coordinates sufficiently overlap, the contour or region1920 (i.e., and the associated object) is not tracked in the frame. Forinstance, as described above, the client 105 may determine whether thedetected contour or region 1920 overlaps at least a threshold percentage(e.g., of 50%) with a region associated with the pixel coordinates(e.g., see overlapping region 1922 of FIG. 19 ). If the criteria of step2114 are satisfied, the client 105 generally, at step 2116, does notdetermine a pixel position for the contour detected at step 2112. Assuch, no pixel position is reported to the server 106, thereby reducingor eliminating the waste of processing resources associated withattempting to track an object when it is not a target object for whichtracking is desired.

Otherwise, if the criteria of step 2114 are satisfied, the client 105determines a pixel position for the contour or region 1920 at step 2118.Determining a pixel position from a contour may involve, for example,(i) determining a region 1920 (e.g., a pixel mask or bounding box)associated with the contour and (ii) determining a centroid or othercharacteristic position of the region as the pixel position. At step2120, the determined pixel position is transmitted to the server 106 tofacilitate global tracking, for example, using predeterminedhomographies, as described elsewhere in this disclosure (e.g., withrespect to FIGS. 24-26 ). For example, the server 106 may receive thedetermined pixel position, access a homography associating pixelcoordinates in images generated by the sensor 108 from which the frameat step 2110 was received to physical coordinates in the space 102, andapply the homography to the pixel coordinates to generate correspondingphysical coordinates for the tracked object associated with the contourdetected at step 2112.

Modifications, additions, or omissions may be made to method 2100depicted in FIG. 21 . Method 2100 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 100,client(s) 105, server 106, or components of any of thereof performingsteps, any suitable system or components of the system may perform oneor more steps of the method.

Contour-Based Detection of Closely Spaced People

In some cases, two people are near each other, making it difficult orimpossible to reliably detect and/or track each person (or other targetobjects) using conventional tools. In some cases, the people may beinitially detected and tracked using depth images at an approximatewaist depth (i.e., a depth corresponding to the waist height of anaverage person being tracked). Tracking at an approximate waist depthmay be more effective at capturing all people regardless of their heightor mode of movement. For instance, by detecting and tacking people at anapproximate waist depth, the tracking system 100 is highly likely todetect tall and short individuals and individuals who may be usingalternative methods of movement (e.g., wheelchairs, and the like).However, if two people with a similar height are standing near eachother, it may be difficult to distinguish between the two people in thetop-view images at the approximate waist depth. Rather than detectingtwo separate people, the tracking system 100 may initially detect thepeople as a single larger object.

This disclosure encompasses the recognition that at a decreased depth(i.e., a depth nearer the heads of the people), the people may be morereadily distinguished. This is because the people's heads are morelikely to be imaged at the decreased depth, and their heads are smallerand less likely to be detected as a single merged region (or contour, asdescribed in greater detail below). As another example, if two peopleenter the space 102 standing close to one another (e.g., holding hands),they may appear to be a single larger object. Since the tracking system100 may initially detect the two people as one person, it may bedifficult to properly identify these people if these people separatewhile in the space 102. As yet another example, if two people whobriefly stand close together are momentarily “lost” or detected as onlya single, larger object, it may be difficult to correctly identify thepeople after they separate from one another. As described elsewhere inthis disclosure (e.g., with respect to FIGS. 19-21 and 24-26 ), people(e.g., the people in the example scenarios described above) may betracked by detecting contours in top-view image frames generated bysensors 108 and tracking the positions of these contours. However, whentwo people are closely spaced, a single merged contour (see mergedcontour 2220 of FIG. 22 described below) may be detected in a top-viewimage of the people. This single contour generally cannot be used totrack each person individually, resulting in considerable downstreamerrors during tracking. For example, even if two people separate afterhaving been closely spaced, it may be difficult or impossible usingprevious tools to determine which person was which, and the identity ofeach person may be unknown after the two people separate. Prior to thisdisclosure, there was a lack of reliable tools for detecting people(e.g., and other target objects) under the example scenarios describedabove and under other similar circumstances.

The systems and methods described in this disclosure provideimprovements to previous technology by facilitating the improveddetection of closely spaced people. For example, the systems and methodsdescribed in this disclosure may facilitate the detection of individualpeople when contours associated with these people would otherwise bemerged, resulting in the detection of a single person using conventionaldetection strategies. In some embodiments, improved contour detection isachieved by detecting contours at different depths (e.g., at least twodepths) to identify separate contours at a second depth within a largermerged contour detected at a first depth used for tracking. For example,if two people are standing near each other such that contours are mergedto form a single contour, separate contours associated with heads of thetwo closely spaced people may be detected at a depth associated with thepersons' heads. In some embodiments, a unique statistical approach maybe used to differentiate between the two people by selecting boundingregions for the detected contours with a low similarity value. In someembodiments, certain criteria are satisfied to ensure that the detectedcontours correspond to separate people, thereby providing more reliableperson (e.g., or other target object) detection than was previouslypossible. For example, two contours detected at an approximate headdepth may be required to be within a threshold size range in order forthe contours to be used for subsequent tracking. In some embodiments, anartificial neural network may be employed to detect separate people thatare closely spaced by analyzing top-view images at different depths.

FIG. 22 is a diagram illustrating the detection of two closely spacedpeople 2202, 2204 based on top-view depth images 2212 and angled-viewimages 2214 received from sensors 108 a,b using the tracking system 100.In one embodiment, sensors 108 a,b may each be one of sensors 108 oftracking system 100 described above with respect to FIG. 1 . In anotherembodiment, sensors 108 a,b may each be one of sensors 108 of a separatevirtual store system (e.g., layout cameras and/or rack cameras) asdescribed in U.S. patent application Ser. No. 16/664,470 entitled,“Customer-Based Video Feed” which is incorporated by reference herein.In this embodiment, the sensors 108 of the tracking system 100 may bemapped to the sensors 108 of the virtual store system using ahomography. Moreover, this embodiment can retrieve identifiers and therelative position of each person from the sensors 108 of the virtualstore system using the homography between tracking system 100 and thevirtual store system. Generally, sensor 108 a is an overhead sensorconfigured to generate top-view depth images 2212 (e.g., color and/ordepth images) of at least a portion of the space 102. Sensor 108 a maybe mounted, for example, in a ceiling of the space 102. Sensor 108 a maygenerate image data corresponding to a plurality of depths which includebut are not necessarily limited to the depths 2210 a-c illustrated inFIG. 22 . Depths 2210 a-c are generally distances measured from thesensor 108 a. Each depth 2210 a-c may be associated with a correspondingheight (e.g., from the floor of the space 102 in which people 2202, 2204are detected and/or tracked). Sensor 108 a observes a field-of-view 2208a. Top-view images 2212 generated by sensor 108 a may be transmitted tothe sensor client 105 a. The sensor client 105 a is communicativelycoupled (e.g., via wired connection or wirelessly) to the sensor 108 aand the server 106. Server 106 is described above with respect to FIG. 1.

In this example, sensor 108 b is an angled-view sensor, which isconfigured to generate angled-view images 2214 (e.g., color and/or depthimages) of at least a portion of the space 102. Sensor 108 b has a fieldof view 2208 b, which overlaps with at least a portion of thefield-of-view 2208 a of sensor 108 a. The angled-view images 2214generated by the angled-view sensor 108 b are transmitted to sensorclient 105 b. Sensor client 105 b may be a client 105 described abovewith respect to FIG. 1 . In the example of FIG. 22 , sensors 108 a,b arecoupled to different sensor clients 105 a,b. However, it should beunderstood that the same sensor client 105 may be used for both sensors108 a,b (e.g., such that clients 105 a,b are the same client 105). Insome cases, the use of different sensor clients 105 a,b for sensors 108a,b may provide improved performance because image data may still beobtained for the area shared by fields-of-view 2208 a,b even if one ofthe clients 105 a,b were to fail.

In the example scenario illustrated in FIG. 22 , people 2202, 2204 arelocated sufficiently close together such that conventional objectdetection tools fail to detect the individual people 2202, 2204 (e.g.,such that people 2202, 2204 would not have been detected as separateobjects). This situation may correspond, for example, to the distance2206 a between people 2202, 2204 being less than a threshold distance2206 b (e.g., of about 6 inches). The threshold distance 2206 b cangenerally be any appropriate distance determined for the system 100. Forexample, the threshold distance 2206 b may be determined based onseveral characteristics of the system 2200 and the people 2202, 2204being detected. For example, the threshold distance 2206 b may be basedon one or more of the distance of the sensor 108 a from the people 2202,2204, the size of the people 2202, 2204, the size of the field-of-view2208 a, the sensitivity of the sensor 108 a, and the like. Accordingly,the threshold distance 2206 b may range from just over zero inches toover six inches depending on these and other characteristics of thetracking system 100. People 2202, 2204 may be any target object anindividual may desire to detect and/or track based on data (i.e.,top-view images 2212 and/or angled-view images 2214) from sensors 108a,b.

The sensor client 105 a detects contours in top-view images 2212received from sensor 108 a. Typically, the sensor client 105 a detectscontours at an initial depth 2210 a. The initial depth 2210 a may beassociated with, for example, a predetermined height (e.g., from theground) which has been established to detect and/or track people 2202,2204 through the space 102. For example, for tracking humans, theinitial depth 2210 a may be associated with an average shoulder or waistheight of people expected to be moving in the space 102 (e.g., a depthwhich is likely to capture a representation for both tall and shortpeople traversing the space 102). The sensor client 105 a may use thetop-view images 2212 generated by sensor 108 a to identify the top-viewimage 2212 corresponding to when a first contour 2202 a associated withthe first person 2202 merges with a second contour 2204 a associatedwith the second person 2204. View 2216 illustrates contours 2202 a, 2204a at a time prior to when these contours 2202 a, 2204 a merge (i.e.,prior to a time (t_(close)) when the first and second people 2202, 2204are within the threshold distance 2206 b of each other). View 2216corresponds to a view of the contours detected in a top-view image 2212received from sensor 108 a (e.g., with other objects in the image notshown).

A subsequent view 2218 corresponds to the image 2212 at or neart_(close) when the people 2202, 2204 are closely spaced and the firstand second contours 2202 a, 2204 a merge to form merged contour 2220.The sensor client 105 a may determine a region 2222 which corresponds toa “size” of the merged contour 2220 in image coordinates (e.g., a numberof pixels associated with contour 2220). For example, region 2222 maycorrespond to a pixel mask or a bounding box determined for contour2220. Example approaches to determining pixel masks and bounding boxesare described above with respect to step 2104 of FIG. 21 . For example,region 2222 may be a bounding box determined for the contour 2220 usinga non-maximum suppression object-detection algorithm. For instance, thesensor client 105 a may determine a plurality of bounding boxesassociated with the contour 2220. For each bounding box, the client 105a may calculate a score. The score, for example, may represent an extentto which that bounding box is similar to the other bounding boxes. Thesensor client 105 a may identify a subset of the bounding boxes with ascore that is greater than a threshold value (e.g., 80% or more), anddetermine region 2222 based on this identified subset. For example,region 2222 may be the bounding box with the highest score or a boundingcomprising regions shared by bounding boxes with a score that is abovethe threshold value.

In order to detect the individual people 2202 and 2204, the sensorclient 105 a may access images 2212 at a decreased depth (i.e., at oneor both of depths 2212 b and 2212 c) and use this data to detectseparate contours 2202 b, 2204 b, illustrated in view 2224. In otherwords, the sensor client 105 a may analyze the images 2212 at a depthnearer the heads of people 2202, 2204 in the images 2212 in order todetect the separate people 2202, 2204. In some embodiments, thedecreased depth may correspond to an average or predetermined headheight of persons expected to be detected by the tracking system 100 inthe space 102. In some cases, contours 2202 b, 2204 b may be detected atthe decreased depth for both people 2202, 2204.

However, in other cases, the sensor client 105 a may not detect bothheads at the decreased depth. For example, if a child and an adult areclosely spaced, only the adult's head may be detected at the decreaseddepth (e.g., at depth 2210 b). In this scenario, the sensor client 105 amay proceed to a slightly increased depth (e.g., to depth 2210 c) todetect the head of the child. For instance, in such scenarios, thesensor client 105 a iteratively increases the depth from the decreaseddepth towards the initial depth 2210 a in order to detect two distinctcontours 2202 b, 2204 b (e.g., for both the adult and the child in theexample described above). For instance, the depth may first be decreasedto depth 2210 b and then increased to depth 2210 c if both contours 2202b and 2204 b are not detected at depth 2210 b. This iterative process isdescribed in greater detail below with respect to method 2300 of FIG. 23.

As described elsewhere in this disclosure, in some cases, the trackingsystem 100 may maintain a record of features, or descriptors, associatedwith each tracked person (see, e.g., FIG. 30 , described below). Assuch, the sensor client 105 a may access this record to determine uniquedepths that are associated with the people 2202, 2204, which are likelyassociated with merged contour 2220. For instance, depth 2210 b may beassociated with a known head height of person 2202, and depth 2212 c maybe associated with a known head height of person 2204.

Once contours 2202 b and 2204 b are detected, the sensor clientdetermines a region 2202 c associated with pixel coordinates 2202 d ofcontour 2202 b and a region 2204 c associated with pixel coordinates2204 d of contour 2204 b. For example, as described above with respectto region 2222, regions 2202 c and 2204 c may correspond to pixel masksor bounding boxes generated based on the corresponding contours 2202 b,2204 b, respectively. For example, pixel masks may be generated to “fillin” the area inside the contours 2202 b, 2204 b or bounding boxes may begenerated which encompass the contours 2202 b, 2204 b. The pixelcoordinates 2202 d, 2204 d generally correspond to the set of positions(e.g., rows and columns) of pixels within regions 2202 c, 2204 c.

In some embodiments, a unique approach is employed to more reliablydistinguish between closely spaced people 2202 and 2204 and determineassociated regions 2202 c and 2204 c. In these embodiments, the regions2202 c and 2204 c are determined using a unique method referred to inthis disclosure as “non-minimum suppression.” Non-minimum suppressionmay involve, for example, determining bounding boxes associated with thecontour 2202 b, 2204 b (e.g., using any appropriate object detectionalgorithm as appreciated by a person of skilled in the relevant art).For each bounding box, a score may be calculated. As described abovewith respect to non-maximum suppression, the score may represent anextent to which the bounding box is similar to the other bounding boxes.However, rather than identifying bounding boxes with high scores (e.g.,as with non-maximum suppression), a subset of the bounding boxes isidentified with scores that are less than a threshold value (e.g., ofabout 20%). This subset may be used to determine regions 2202 c, 2204 c.For example, regions 2202 c, 2204 c may include regions shared by eachbounding box of the identified subsets. In other words, bounding boxesthat are not below the minimum score are “suppressed” and not used toidentify regions 2202 b, 2204 b.

Prior to assigning a position or identity to the contours 2202 b, 2204 band/or the associated regions 2202 c, 2204 c, the sensor client 105 amay first check whether criteria are satisfied for distinguishing theregion 2202 c from region 2204 c. The criteria are generally designed toensure that the contours 2202 b, 2204 b (and/or the associated regions2202 c, 2204 c) are appropriately sized, shaped, and positioned to beassociated with the heads of the corresponding people 2202, 2204. Thesecriteria may include one or more requirements. For example, onerequirement may be that the regions 2202 c, 2204 c overlap by less thanor equal to a threshold amount (e.g., of about 50%, e.g., of about 10%).Generally, the separate heads of different people 2202, 2204 should notoverlap in a top-view image 2212. Another requirement may be that theregions 2202 c, 2204 c are within (e.g., bounded by, e.g., encompassedby) the merged-contour region 2222. This requirement, for example,ensures that the head contours 2202 b, 2204 b are appropriatelypositioned above the merged contour 2220 to correspond to heads ofpeople 2202, 2204. If the contours 2202 b, 2204 b detected at thedecreased depth are not within the merged contour 2220, then thesecontours 2202 b, 2204 b are likely not associated with heads of thepeople 2202, 2204 associated with the merged contour 2220.

Generally, if the criteria are satisfied, the sensor client 105 aassociates region 2202 c with a first pixel position 2202 e of person2202 and associates region 2204 c with a second pixel position 2204 e ofperson 2204. Each of the first and second pixel positions 2202 e, 2204 egenerally corresponds to a single pixel position (e.g., row and column)associated with the location of the corresponding contour 2202 b, 2204 bin the image 2212. The first and second pixel positions 2202 e, 2204 eare included in the pixel positions 2226 which may be transmitted to theserver 106 to determine corresponding physical (e.g., global) positions2228, for example, based on homographies 2230 (e.g., using a previouslydetermined homography for sensor 108 a associating pixel coordinates inimages 2212 generated by sensor 108 a to physical coordinates in thespace 102).

As described above, sensor 108 b is positioned and configured togenerate angled-view images 2214 of at least a portion of the fieldof-of-view 2208 a of sensor 108 a. The sensor client 105 b receives theangled-view images 2214 from the second sensor 108 b. Because of itsdifferent (e.g., angled) view of people 2202, 2204 in the space 102, anangled-view image 2214 obtained at t_(close) may be sufficient todistinguish between the people 2202, 2204. A view 2232 of contours 2202d, 2204 d detected at t_(close) is shown in FIG. 22 . The sensor client105 b detects a contour 2202 f corresponding to the first person 2202and determines a corresponding region 2202 g associated with pixelcoordinates 2202 h of contour 2202 f The sensor client 105 b detects acontour 2204 f corresponding to the second person 2204 and determines acorresponding region 2204 g associated with pixel coordinates 2204 h ofcontour 2204 f. Since contours 2202 f, 2204 f do not merge and regions2202 g, 2204 g are sufficiently separated (e.g., they do not overlapand/or are at least a minimum pixel distance apart), the sensor client105 b may associate region 2202 g with a first pixel position 2202 i ofthe first person 2202 and region 2204 g with a second pixel position2204 i of the second person 2204. Each of the first and second pixelpositions 2202 i, 2204 i generally corresponds to a single pixelposition (e.g., row and column) associated with the location of thecorresponding contour 2202 f, 2204 f in the image 2214. Pixel positions2202 i, 2204 i may be included in pixel positions 2234 which may betransmitted to server 106 to determine physical positions 2228 of thepeople 2202, 2204 (e.g., using a previously determined homography forsensor 108 b associating pixel coordinates of images 2214 generated bysensor 108 b to physical coordinates in the space 102).

In an example operation of the tracking system 100, sensor 108 a isconfigured to generate top-view color-depth images of at least a portionof the space 102. When people 2202 and 2204 are within a thresholddistance of each another, the sensor client 105 a identifies an imageframe (e.g., associated with view 2218) corresponding to a time stamp(e.g., t_(close)) where contours 2202 a, 2204 a associated with thefirst and second person 2202, 2204, respectively, are merged and formcontour 2220. In order to detect each person 2202 and 2204 in theidentified image frame (e.g., associated with view 2218), the client 105a may first attempt to detect separate contours for each person 2202,2204 at a first decreased depth 2210 b. As described above, depth 2210 bmay be a predetermined height associated with an expected head height ofpeople moving through the space 102. In some embodiments, depth 2210 bmay be a depth previously determined based on a measured height ofperson 2202 and/or a measured height of person 2204. For example, depth2210 b may be based on an average height of the two people 2202, 2204.As another example, depth 2210 b may be a depth corresponding to apredetermined head height of person 2202 (as illustrated in the exampleof FIG. 22 ). If two contours 2202 b, 2204 b are detected at depth 2210b, these contours may be used to determine pixel positions 2202 e, 2204e of people 2202 and 2204, as described above.

If only one contour 2202 b is detected at depth 2210 b (e.g., if onlyone person 2202, 2204 is tall enough to be detected at depth 2210 b),the region associated with this contour 2202 b may be used to determinethe pixel position 2202 e of the corresponding person, and the nextperson may be detected at an increased depth 2210 c. Depth 2210 c isgenerally greater than 2210 b but less than depth 2210 a. In theillustrative example of FIG. 22 , depth 2210 c corresponds to apredetermined head height of person 2204. If contour 2204 b is detectedfor person 2204 at depth 2210 c, a pixel position 2204 e is determinedbased on pixel coordinates 2204 d associated with the contour 2204 b(e.g., following a determination that the criteria described above aresatisfied). If a contour 2204 b is not detected at depth 2210 c, theclient 105 a may attempt to detect contours at progressively increaseddepths until a contour is detected or a maximum depth (e.g., the initialdepth 2210 a) is reached. For example, the sensor client 105 a maycontinue to search for the contour 2204 b at increased depths (i.e.,depths between depth 2210 c and the initial depth 2210 a). If themaximum depth (e.g., depth 2210 a) is reached without the contour 2204 bbeing detected, the client 105 a generally determines that the separatepeople 2202, 2204 cannot be detected.

FIG. 23 is a flowchart illustrating a method 2300 of operating thetracking system 100 to detect closely spaced people 2202, 2204. Method2300 may begin at step 2302 where the sensor client 105 a receives oneor more frames of top-view depth images 2212 generated by sensor 108 a.At step 2304, the sensor client 105 a identifies a frame in which afirst contour 2202 a associated with the first person 2202 is mergedwith a second contour 2204 a associated with the second person 2204.Generally, the merged first and second contours (i.e., merged contour2220) is determined at the first depth 2212 a in the depth images 2212received at step 2302. The first depth 2212 a may correspond to a waistor should depth of persons expected to be tracked in the space 102. Thedetection of merged contour 2220 corresponds to the first person 2202being located in the space within a threshold distance 2206 b from thesecond person 2204, as described above.

At step 2306, the sensor client 105 a determines a merged-contour region2222. Region 2222 is associated with pixel coordinates of the mergedcontour 2220. For instance, region 2222 may correspond to coordinates ofa pixel mask that overlays the detected contour. As another example,region 2222 may correspond to pixel coordinates of a bounding boxdetermined for the contour (e.g., using any appropriate object detectionalgorithm). In some embodiments, a method involving non-maximumsuppression is used to detect region 2222. In some embodiments, region2222 is determined using an artificial neural network. For example, anartificial neural network may be trained to detect contours at variousdepths in top-view images generated by sensor 108 a.

At step 2308, the depth at which contours are detected in the identifiedimage frame from step 2304 is decreased (e.g., to depth 2210 billustrated in FIG. 22 ). At step 2310 a, the sensor client 105 adetermines whether a first contour (e.g., contour 2202 b) is detected atthe current depth. If the contour 2202 b is not detected, the sensorclient 105 a proceeds, at step 2312 a, to an increased depth (e.g., todepth 2210 c). If the increased depth corresponds to having reached amaximum depth (e.g., to reaching the initial depth 2210 a), the processends because the first contour 2202 b was not detected. If the maximumdepth has not been reached, the sensor client 105 a returns to step 2310a and determines if the first contour 2202 b is detected at the newlyincreased current depth. If the first contour 2202 b is detected at step2310 a, the sensor client 105 a, at step 2316 a, determines a firstregion 2202 c associated with pixel coordinates 2202 d of the detectedcontour 2202 b. In some embodiments, region 2202 c may be determinedusing a method of non-minimal suppression, as described above. In someembodiments, region 2202 c may be determined using an artificial neuralnetwork.

The same or a similar approach—illustrated in steps 2210 b, 2212 b, 2214b, and 2216 b—may be used to determine a second region 2204 c associatedwith pixel coordinates 2204 d of the contour 2204 b. For example, atstep 2310 b, the sensor client 105 a determines whether a second contour2204 b is detected at the current depth. If the contour 2204 b is notdetected, the sensor client 105 a proceeds, at step 2312 b, to anincreased depth (e.g., to depth 2210 c). If the increased depthcorresponds to having reached a maximum depth (e.g., to reaching theinitial depth 2210 a), the process ends because the second contour 2204b was not detected. If the maximum depth has not been reached, thesensor client 105 a returns to step 2310 b and determines if the secondcontour 2204 b is detected at the newly increased current depth. If thesecond contour 2204 b is detected at step 2210 a, the sensor client 105a, at step 2316 a, determines a second region 2204 c associated withpixel coordinates 2204 d of the detected contour 2204 b. In someembodiments, region 2204 c may be determined using a method ofnon-minimal suppression or an artificial neural network, as describedabove.

At step 2318, the sensor client 105 a determines whether criteria aresatisfied for distinguishing the first and second regions determined insteps 2316 a and 2316 b, respectively. For example, the criteria mayinclude one or more requirements. For example, one requirement may bethat the regions 2202 c, 2204 c overlap by less than or equal to athreshold amount (e.g., of about 10%). Another requirement may be thatthe regions 2202 c, 2204 c are within (e.g., bounded by, e.g.,encompassed by) the merged-contour region 2222 (determined at step2306). If the criteria are not satisfied, method 2300 generally ends.

Otherwise, if the criteria are satisfied at step 2318, the method 2300proceeds to steps 2320 and 2322 where the sensor client 105 a associatesthe first region 2202 b with a first pixel position 2202 e of the firstperson 2202 (step 2320) and associates the second region 2204 b with afirst pixel position 2202 e of the first person 2204 (step 2322).Associating the regions 2202 c, 2204 c to pixel positions 2202 e, 2204 emay correspond to storing in a memory pixel coordinates 2202 d, 2204 dof the regions 2202 c, 2204 c and/or an average pixel positioncorresponding to each of the regions 2202 c, 2204 c along with an objectidentifier for the people 2202, 2204.

At step 2324, the sensor client 105 a may transmit the first and secondpixel positions (e.g., as pixel positions 2226) to the server 106. Atstep 2326, the server 106 may apply a homography (e.g., of homographies2230) for the sensor 2202 to the pixel positions to determinecorresponding physical (e.g., global) positions 2228 for the first andsecond people 2202, 2204. Examples of generating and using homographies2230 are described in greater detail above with respect to FIGS. 2-7 .

Modifications, additions, or omissions may be made to method 2300depicted in FIG. 23 . Method 2300 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as system 2200, sensor client22105 a, master server 2208, or components of any of thereof performingsteps, any suitable system or components of the system may perform oneor more steps of the method.

Multi-Sensor Image Tracking on a Local and Global Planes

As described elsewhere in this disclosure (e.g., with respect to FIGS.19-23 above), tracking people (e.g., or other target objects) in space102 using multiple sensors 108 presents several previously unrecognizedchallenges. This disclosure encompasses not only the recognition ofthese challenges but also unique solutions to these challenges. Forinstance, systems and methods are described in this disclosure thattrack people both locally (e.g., by tracking pixel positions in imagesreceived from each sensor 108) and globally (e.g., by tracking physicalpositions on a global plane corresponding to the physical coordinates inthe space 102). Person tracking may be more reliable when performed bothlocally and globally. For example, if a person is “lost” locally (e.g.,if a sensor 108 fails to capture a frame and a person is not detected bythe sensor 108), the person may still be tracked globally based on animage from a nearby sensor 108 (e.g., the angled-view sensor 108 bdescribed with respect to FIG. 22 above), an estimated local position ofthe person determined using a local tracking algorithm, and/or anestimated global position determined using a global tracking algorithm.

As another example, if people appear to merge (e.g., if detectedcontours merge into a single merged contour, as illustrated in view 2216of FIG. 22 above) at one sensor 108, an adjacent sensor 108 may stillprovide a view in which the people are separate entities (e.g., asillustrated in view 2232 of FIG. 22 above). Thus, information from anadjacent sensor 108 may be given priority for person tracking. In someembodiments, if a person tracked via a sensor 108 is lost in the localview, estimated pixel positions may be determined using a trackingalgorithm and reported to the server 106 for global tracking, at leastuntil the tracking algorithm determines that the estimated positions arebelow a threshold confidence level.

FIGS. 24A-C illustrate the use of a tracking subsystem 2400 to track aperson 2402 through the space 102. FIG. 24A illustrates a portion of thetracking system 100 of FIG. 1 when used to track the position of person2402 based on image data generated by sensors 108 a-c. The position ofperson 2402 is illustrated at three different time points: t₁, t₂, andt₃. Each of the sensors 108 a-c is a sensor 108 of FIG. 1 , describedabove. Each sensor 108 a-c has a corresponding field-of-view 2404 a-c,which corresponds to the portion of the space 102 viewed by the sensor108 a-c. As shown in FIG. 24A, each field-of-view 2404 a-c overlaps withthat of the adjacent sensor(s) 108 a-c. For example, the adjacentfields-of-view 2404 a-c may overlap by between about 10% and 30%.Sensors 108 a-c generally generate top-view images and transmitcorresponding top-view image feeds 2406 a-c to a tracking subsystem2400.

The tracking subsystem 2400 includes the client(s) 105 and server 106 ofFIG. 1 . The tracking system 2400 generally receives top-view imagefeeds 2406 a-c generated by sensors 108 a-c, respectively, and uses thereceived images (see FIG. 24B) to track a physical (e.g., global)position of the person 2402 in the space 102 (see FIG. 24C). Each sensor108 a-c may be coupled to a corresponding sensor client 105 of thetracking subsystem 2400. As such, the tracking subsystem 2400 mayinclude local particle filter trackers 2444 for tracking pixel positionsof person 2402 in images generated by sensors 108 a-b, global particlefilter trackers 2446 for tracking physical positions of person 2402 inthe space 102.

FIG. 24B shows example top-view images 2408 a-c, 2418 a-c, and 2426 a-cgenerated by each of the sensors 108 a-c at times t₁, t₂, and t₃.Certain of the top-view images include representations of the person2402 (i.e., if the person 2402 was in the field-of-view 2404 a-c of thesensor 108 a-c at the time the image 2408 a-c, 2418 a-c, and 2426 a-cwas obtained). For example, at time t₁, images 2408 a-c are generated bysensors 108 a-c, respectively, and provided to the tracking subsystem2400. The tracking subsystem 2400 detects a contour 2410 associated withperson 2402 in image 2408 a. For example, the contour 2410 maycorrespond to a curve outlining the border of a representation of theperson 2402 in image 2408 a (e.g., detected based on color (e.g., RGB)image data at a predefined depth in image 2408 a, as described abovewith respect to FIG. 19 ). The tracking subsystem 2400 determines pixelcoordinates 2412 a, which are illustrated in this example by thebounding box 2412 b in image 2408 a. Pixel position 2412 c is determinedbased on the coordinates 2412 a. The pixel position 2412 c generallyrefers to the location (i.e., row and column) of the person 2402 in theimage 2408 a. Since the object 2402 is also within the field-of-view2404 b of the second sensor 108 b at t₁ (see FIG. 24A), the trackingsystem also detects a contour 2414 in image 2408 b and determinescorresponding pixel coordinates 2416 a (i.e., associated with boundingbox 2416 b) for the object 2402. Pixel position 2416 c is determinedbased on the coordinates 2416 a. The pixel position 2416 c generallyrefers to the pixel location (i.e., row and column) of the person 2402in the image 2408 b. At time t₁, the object 2402 is not in thefield-of-view 2404 c of the third sensor 108 c (see FIG. 24A).Accordingly, the tracking subsystem 2400 does not determine pixelcoordinates for the object 2402 based on the image 2408 c received fromthe third sensor 108 c.

Turning now to FIG. 24C, the tracking subsystem 2400 (e.g., the server106 of the tacking subsystem 2400) may determine a first global position2438 based on the determined pixel positions 2412 c and 2416 c (e.g.,corresponding to pixel coordinates 2412 a, 2416 a and bounding boxes2412 b, 2416 b, described above). The first global position 2438corresponds to the position of the person 2402 in the space 102, asdetermined by the tracking subsystem 2400. In other words, the trackingsubsystem 2400 uses the pixel positions 2412 c, 2416 c determined viathe two sensors 108 a,b to determine a single physical position 2438 forthe person 2402 in the space 102. For example, a first physical position2412 d may be determined from the pixel position 2412 c associated withbounding box 2412 b using a first homography associating pixelcoordinates in the top-view images generated by the first sensor 108 ato physical coordinates in the space 102. A second physical position2416 d may similarly be determined using the pixel position 2416 cassociated with bounding box 2416 b using a second homographyassociating pixel coordinates in the top-view images generated by thesecond sensor 108 b to physical coordinates in the space 102. In somecases, the tracking subsystem 2400 may compare the distance betweenfirst and second physical positions 2412 d and 2416 d to a thresholddistance 2448 to determine whether the positions 2412 d, 2416 dcorrespond to the same person or different people (see, e.g., step 2620of FIG. 26 , described below). The first global position 2438 may bedetermined as an average of the first and second physical positions 2410d, 2414 d. In some embodiments, the global position is determined byclustering the first and second physical positions 2410 d, 2414 d (e.g.,using any appropriate clustering algorithm). The first global position2438 may correspond to (x,y) coordinates of the position of the person2402 in the space 102.

Returning to FIG. 24A, at time t₂, the object 2402 is withinfields-of-view 2404 a and 2404 b corresponding to sensors 108 a,b. Asshown in FIG. 24B, a contour 2422 is detected in image 2418 b andcorresponding pixel coordinates 2424 a, which are illustrated bybounding box 2424 b, are determined. Pixel position 2424 c is determinedbased on the coordinates 2424 a. The pixel position 2424 c generallyrefers to the location (i.e., row and column) of the person 2402 in theimage 2418 b. However, in this example, the tracking subsystem 2400fails to detect, in image 2418 a from sensor 108 a, a contour associatedwith object 2402. This may be because the object 2402 was at the edge ofthe field-of-view 2404 a, because of a lost image frame from feed 2406a, because the position of the person 2402 in the field-of-view 2404 acorresponds to an auto-exclusion zone for sensor 108 a (see FIGS. 19-21and corresponding description above), or because of any othermalfunction of sensor 108 a and/or the tracking subsystem 2400. In thiscase, the tracking subsystem 2400 may locally (e.g., at the particularclient 105 which is coupled to sensor 108 a) estimate pixel coordinates2420 a and/or corresponding pixel position 2420 b for object 2402. Forexample, a local particle filter tracker 2444 for object 2402 in imagesgenerated by sensor 108 a may be used to determine the estimated pixelposition 2420 b.

FIGS. 25A,B illustrate the operation of an example particle filtertracker 2444, 2446 (e.g., for determining estimated pixel position 2420a). FIG. 25A illustrates a region 2500 in pixel coordinates or physicalcoordinates of space 102. For example, region 2500 may correspond to apixel region in an image or to a region in physical space. In a firstzone 2502, an object (e.g., person 2402) is detected at position 2504.The particle filter determines several estimated subsequent positions2506 for the object. The estimated subsequent positions 2506 areillustrated as the dots or “particles” in FIG. 25A and are generallydetermined based on a history of previous positions of the object.Similarly, another zone 2508 shows a position 2510 for another object(or the same object at a different time) along with estimated subsequentpositions 2512 of the “particles” for this object.

For the object at position 2504, the estimated subsequent positions 2506are primarily clustered in a similar area above and to the right ofposition 2504, indicating that the particle filter tracker 2444, 2446may provide a relatively good estimate of a subsequent position.Meanwhile, the estimated subsequent positions 2512 are relativelyrandomly distributed around position 2510 for the object, indicatingthat the particle filter tracker 2444, 2446 may provide a relativelypoor estimate of a subsequent position. FIG. 25B shows a distributionplot 2550 of the particles illustrated in FIG. 25A, which may be used toquantify the quality of an estimated position based on a standarddeviation value (σ).

In FIG. 25B, curve 2552 corresponds to the position distribution ofanticipated positions 2506, and curve 2554 corresponds to the positiondistribution of the anticipated positions 2512. Curve 2554 has arelatively narrow distribution such that the anticipated positions 2506are primarily near the mean position (μ). For example, the narrowdistribution corresponds to the particles primarily having a similarposition, which in this case is above and to right of position 2504. Incontrast, curve 2554 has a broader distribution, where the particles aremore randomly distributed around the mean position (O. Accordingly, thestandard deviation of curve 2552 (σ₁) is smaller than the standarddeviation curve 2554 (σ₂). Generally, a standard deviation (e.g., eitheral or σ₂) may be used as a measure of an extent to which an estimatedpixel position generated by the particle filter tracker 2444, 2446 islikely to be correct. If the standard deviation is less than a thresholdstandard deviation (σ_(threshold)), as is the case with curve 2552 andσ₁, the estimated position generated by a particle filter tracker 2444,2446 may be used for object tracking. Otherwise, the estimated positiongenerally is not used for object tracking.

Referring again to FIG. 24C, the tracking subsystem 2400 (e.g., theserver 106 of tracking subsystem 2400) may determine a second globalposition 2440 for the object 2402 in the space 102 based on theestimated pixel position 2420 b associated with estimated bounding box2420 a in frame 2418 a and the pixel position 2424 c associated withbounding box 2424 b from frame 2418 b. For example, a first physicalposition 2420 c may be determined using a first homography associatingpixel coordinates in the top-view images generated by the first sensor108 a to physical coordinates in the space 102. A second physicalposition 2424 d may be determined using a second homography associatingpixel coordinates in the top-view images generated by the second sensor108 b to physical coordinates in the space 102. The tracking subsystem2400 (i.e., server 106 of the tracking subsystem 2400) may determine thesecond global position 2440 based on the first and second physicalpositions 2420 c, 2424 d, as described above with respect to time t₁.The second global position 2440 may correspond to (x,y) coordinates ofthe person 2402 in the space 102.

Turning back to FIG. 24A, at time t₃, the object 2402 is within thefield-of-view 2404 b of sensor 108 b and the field-of-view 2404 c ofsensor 108 c. Accordingly, these images 2426 b,c may be used to trackperson 2402. FIG. 24B shows that a contour 2428 and corresponding pixelcoordinates 2430 a, pixel region 2430 b, and pixel position 2430 c aredetermined in frame 2426 b from sensor 108 b, while a contour 2432 andcorresponding pixel coordinates 2434 a, pixel region 2434 b, and pixelposition 2434 c are detected in frame 2426 c from sensor 108 c. As shownin FIG. 24C and as described in greater detail above for times t₁ andt₂, the tracking subsystem 2400 may determine a third global position2442 for the object 2402 in the space based on the pixel position 2430 cassociated with bounding box 2430 b in frame 2426 b and the pixelposition 2434 c associated with bounding box 2434 b from frame 2426 c.For example, a first physical position 2430 d may be determined using asecond homography associating pixel coordinates in the top-view imagesgenerated by the second sensor 108 b to physical coordinates in thespace 102. A second physical position 2434 d may be determined using athird homography associating pixel coordinates in the top-view imagesgenerated by the third sensor 108 c to physical coordinates in the space102. The tracking subsystem 2400 may determine the global position 2442based on the first and second physical positions 2430 d, 2434 d, asdescribed above with respect to times t₁ and t₂.

FIG. 26 is a flow diagram illustrating the tracking of person 2402 inspace the 102 based on top-view images (e.g., images 2408 a-c, 2418 a 0c, 2426 a-c from feeds 2406 a,b, generated by sensors 108 a,b, describedabove. Field-of-view 2404 a of sensor 108 a and field-of-view 2404 b ofsensors 108 b generally overlap by a distance 2602. In one embodiment,distance 2602 may be about 10% to 30% of the fields-of-view 2404 a,b. Inthis example, the tracking subsystem 2400 includes the first sensorclient 105 a, the second sensor client 105 b, and the server 106. Eachof the first and second sensor clients 105 a,b may be a client 105described above with respect to FIG. 1 . The first sensor client 105 ais coupled to the first sensor 108 a and configured to track, based onthe first feed 2406 a, a first pixel position 2112 c of the person 2402.The second sensor client 105 b is coupled to the second sensor 108 b andconfigured to track, based on the second feed 2406 b, a second pixelposition 2416 c of the same person 2402.

The server 106 generally receives pixel positions from clients 105 a,band tracks the global position of the person 2402 in the space 102. Insome embodiments, the server 106 employs a global particle filtertracker 2446 to track a global physical position of the person 2402 andone or more other people 2604 in the space 102). Tracking people bothlocally (i.e., at the “pixel level” using clients 105 a,b) and globally(i.e., based on physical positions in the space 102) improves trackingby reducing and/or eliminating noise and/or other tracking errors whichmay result from relying on either local tracking by the clients 105 a,bor global tracking by the server 106 alone.

FIG. 26 illustrates a method 2600 implemented by sensor clients 105 a,band server 106. Sensor client 105 a receives the first data feed 2406 afrom sensor 108 a at step 2606 a. The feed may include top-view images(e.g., images 2408 a-c, 2418 a-c, 2426 a-c of FIG. 24 ). The images maybe color images, depth images, or color-depth images. In an image fromthe feed 2406 a (e.g., corresponding to a certain timestamp), the sensorclient 105 a determines whether a contour is detected at step 2608 a. Ifa contour is detected at the timestamp, the sensor client 105 adetermines a first pixel position 2412 c for the contour at step 2610 a.For instance, the first pixel position 2412 c may correspond to pixelcoordinates associated with a bounding box 2412 b determined for thecontour (e.g., using any appropriate object detection algorithm). Asanother example, the sensor client 105 a may generate a pixel mask thatoverlays the detected contour and determine pixel coordinates of thepixel mask, as described above with respect to step 2104 of FIG. 21 .

If a contour is not detected at step 2608 a, a first particle filtertracker 2444 may be used to estimate a pixel position (e.g., estimatedposition 2420 b), based on a history of previous positions of thecontour 2410, at step 2612 a. For example, the first particle filtertracker 2444 may generate a probability-weighted estimate of asubsequent first pixel position corresponding to the timestamp (e.g., asdescribed above with respect to FIGS. 25A,B). Generally, if theconfidence level (e.g., based on a standard deviation) of the estimatedpixel position 2420 b is below a threshold value (e.g., see FIG. 25B andrelated description above), no pixel position is determined for thetimestamp by the sensor client 105 a, and no pixel position is reportedto server 106 for the timestamp. This prevents the waste of processingresources which would otherwise be expended by the server 106 inprocessing unreliable pixel position data. As described below, theserver 106 can often still track person 2402, even when no pixelposition is provided for a given timestamp, using the global particlefilter tracker 2446 (see steps 2626, 2632, and 2636 below).

The second sensor client 105 b receives the second data feed 2406 b fromsensor 108 b at step 2606 b. The same or similar steps to thosedescribed above for sensor client 105 a are used to determine a secondpixel position 2416 c for a detected contour 2414 or estimate a pixelposition based on a second particle filter tracker 2444. At step 2608 b,the sensor client 105 b determines whether a contour 2414 is detected inan image from feed 2406 b at a given timestamp. If a contour 2414 isdetected at the timestamp, the sensor client 105 b determines a firstpixel position 2416 c for the contour 2414 at step 2610 b (e.g., usingany of the approaches described above with respect to step 2610 a). If acontour 2414 is not detected, a second particle filter tracker 2444 maybe used to estimate a pixel position at step 2612 b (e.g., as describedabove with respect to step 2612 a). If the confidence level of theestimated pixel position is below a threshold value (e.g., based on astandard deviation value for the tracker 2444), no pixel position isdetermined for the timestamp by the sensor client 105 b, and no pixelposition is reported for the timestamp to the server 106.

While steps 2606 a,b-2612 a,b are described as being performed by sensorclient 105 a and 105 b, it should be understood that in someembodiments, a single sensor client 105 may receive the first and secondimage feeds 2406 a,b from sensors 108 a,b and perform the stepsdescribed above. Using separate sensor clients 105 a,b for separatesensors 108 a,b or sets of sensors 108 may provide redundancy in case ofclient 105 malfunctions (e.g., such that even if one sensor client 105fails, feeds from other sensors may be processed by otherstill-functioning clients 105).

At step 2614, the server 106 receives the pixel positions 2412 c, 2416 cdetermined by the sensor clients 105 a,b. At step 2616, the server 106may determine a first physical position 2412 d based on the first pixelposition 2412 c determined at step 2610 a or estimated at step 2612 a bythe first sensor client 105 a. For example, the first physical position2412 d may be determined using a first homography associating pixelcoordinates in the top-view images generated by the first sensor 108 ato physical coordinates in the space 102. At step 2618, the server 106may determine a second physical position 2416 d based on the secondpixel position 2416 c determined at step 2610 b or estimated at step2612 b by the first sensor client 105 b. For instance, the secondphysical position 2416 d may be determined using a second homographyassociating pixel coordinates in the top-view images generated by thesecond sensor 108 b to physical coordinates in the space 102.

At step 2620 the server 106 determines whether the first and secondpositions 2412 d, 2416 d (from steps 2616 and 2618) are within athreshold distance 2448 (e.g., of about six inches) of each other. Ingeneral, the threshold distance 2448 may be determined based on one ormore characteristics of the system tracking system 100 and/or the person2402 or another target object being tracked. For example, the thresholddistance 2448 may be based on one or more of the distance of the sensors108 a-b from the object, the size of the object, the fields-of-view 2404a-b, the sensitivity of the sensors 108 a-b, and the like. Accordingly,the threshold distance 2448 may range from just over zero inches togreater than six inches depending on these and other characteristics ofthe tracking system 100.

If the positions 2412 d, 2416 d are within the threshold distance 2448of each other at step 2620, the server 106 determines that the positions2412 d, 2416 d correspond to the same person 2402 at step 2622. In otherwords, the server 106 determines that the person detected by the firstsensor 108 a is the same person detected by the second sensor 108 b.This may occur, at a given timestamp, because of the overlap 2604between field-of-view 2404 a and field-of-view 2404 b of sensors 108 aand 108 b, as illustrated in FIG. 26 .

At step 2624, the server 106 determines a global position 2438 (i.e., aphysical position in the space 102) for the object based on the firstand second physical positions from steps 2616 and 2618. For instance,the server 106 may calculate an average of the first and second physicalpositions 2412 d, 2416 d. In some embodiments, the global position 2438is determined by clustering the first and second physical positions 2412d, 2416 d (e.g., using any appropriate clustering algorithm). At step2626, a global particle filter tracker 2446 is used to track the global(e.g., physical) position 2438 of the person 2402. An example of aparticle filter tracker is described above with respect to FIGS. 25A,B.For instance, the global particle filter tracker 2446 may generateprobability-weighted estimates of subsequent global positions atsubsequent times. If a global position 2438 cannot be determined at asubsequent timestamp (e.g., because pixel positions are not availablefrom the sensor clients 105 a,b), the particle filter tracker 2446 maybe used to estimate the position.

If at step 2620 the first and second physical positions 2412 d, 2416 dare not within the threshold distance 2448 from each other, the server106 generally determines that the positions correspond to differentobjects 2402, 2604 at step 2628. In other words, the server 106 maydetermine that the physical positions determined at steps 2616 and 2618are sufficiently different, or far apart, for them to correspond to thefirst person 2402 and a different second person 2604 in the space 102.

At step 2630, the server 106 determines a global position for the firstobject 2402 based on the first physical position 2412 c from step 2616.Generally, in the case of having only one physical position 2412 c onwhich to base the global position, the global position is the firstphysical position 2412 c. If other physical positions are associatedwith the first object (e.g., based on data from other sensors 108, whichfor clarity are not shown in FIG. 26 ), the global position of the firstperson 2402 may be an average of the positions or determined based onthe positions using any appropriate clustering algorithm, as describedabove. At step 2632, a global particle filter tracker 2446 may be usedto track the first global position of the first person 2402, as is alsodescribed above.

At step 2634, the server 106 determines a global position for the secondperson 2404 based on the second physical position 2416 c from step 2618.Generally, in the case of having only one physical position 2416 c onwhich to base the global position, the global position is the secondphysical position 2416 c. If other physical positions are associatedwith the second object (e.g., based on data from other sensors 108,which not shown in FIG. 26 for clarity), the global position of thesecond person 2604 may be an average of the positions or determinedbased on the positions using any appropriate clustering algorithm. Atstep 2636, a global particle filter tracker 2446 is used to track thesecond global position of the second object, as described above.

Modifications, additions, or omissions may be made to the method 2600described above with respect to FIG. 26 . The method may include more,fewer, or other steps. For example, steps may be performed in parallelor in any suitable order. While at times discussed as a trackingsubsystem 2400, sensor clients 105 a,b, server 106, or components of anythereof performing steps, any suitable system or components of thesystem may perform one or more steps of the method 2600.

Candidate Lists

When the tracking system 100 is tracking people in the space 102, it maybe challenging to reliably identify people under certain circumstancessuch as when they pass into or near an auto-exclusion zone (see FIGS.19-21 and corresponding description above), when they stand near anotherperson (see FIGS. 22-23 and corresponding description above), and/orwhen one or more of the sensors 108, client(s) 105, and/or server 106malfunction. For instance, after a first person becomes close to or evencomes into contact with (e.g., “collides” with) a second person, it maydifficult to determine which person is which (e.g., as described abovewith respect to FIG. 22 ). Conventional tracking systems may usephysics-based tracking algorithms in an attempt to determine whichperson is which based on estimated trajectories of the people (e.g.,estimated as though the people are marbles colliding and changingtrajectories according to a conservation of momentum, or the like).However, the identities of people may be more difficult to trackreliably, because movements may be random. As described above, thetracking system 100 may employ particle filter tracking for improvedtracking of people in the space 102 (see e.g., FIGS. 24-26 and thecorresponding description above). However, even with these advancements,the identities of people being tracked may be difficult to determine atcertain times. This disclosure particularly encompasses the recognitionthat positions of people who are shopping in a store (i.e., moving abouta space, selecting items, and picking up the items) are difficult orimpossible to track using previously available technology becausemovement of these people is random and does not follow a readily definedpattern or model (e.g., such as the physics-based models of previousapproaches). Accordingly, there is a lack of tools for reliably andefficiently tracking people (e.g., or other target objects).

This disclosure provides a solution to the problems of previoustechnology, including those described above, by maintaining a record,which is referred to in this disclosure as a “candidate list,” ofpossible person identities, or identifiers (i.e., the usernames, accountnumbers, etc. of the people being tracked), during tracking. A candidatelist is generated and updated during tracking to establish the possibleidentities of each tracked person. Generally, for each possible identityor identifier of a tracked person, the candidate list also includes aprobability that the identity, or identifier, is believed to be correct.The candidate list is updated following interactions (e.g., collisions)between people and in response to other uncertainty events (e.g., a lossof sensor data, imaging errors, intentional trickery, etc.).

In some cases, the candidate list may be used to determine when a personshould be re-identified (e.g., using methods described in greater detailbelow with respect to FIGS. 29-32 ). Generally, re-identification isappropriate when the candidate list of a tracked person indicates thatthe person's identity is not sufficiently well known (e.g., based on theprobabilities stored in the candidate list being less than a thresholdvalue). In some embodiments, the candidate list is used to determinewhen a person is likely to have exited the space 102 (i.e., with atleast a threshold confidence level), and an exit notification is onlysent to the person after there is high confidence level that the personhas exited (see, e.g., view 2730 of FIG. 27 , described below). Ingeneral, processing resources may be conserved by only performingpotentially complex person re-identification tasks when a candidate listindicates that a person's identity is no longer known according topre-established criteria.

FIG. 27 is a flow diagram illustrating how identifiers 2701 a-cassociated with tracked people (e.g., or any other target object) may beupdated during tracking over a period of time from an initial time t₀ toa final time t₅ by tracking system 100. People may be tracked usingtracking system 100 based on data from sensors 108, as described above.FIG. 27 depicts a plurality of views 2702, 2716, 2720, 2724, 2728, 2730at different time points during tracking. In some embodiments, views2702, 2716, 2720, 2724, 2728, 2730 correspond to a local frame view(e.g., as described above with respect to FIG. 22 ) from a sensor 108with coordinates in units of pixels (e.g., or any other appropriate unitfor the data type generated by the sensor 108). In other embodiments,views 2702, 2716, 2720, 2724, 2728, 2730 correspond to global views ofthe space 102 determined based on data from multiple sensors 108 withcoordinates corresponding to physical positions in the space (e.g., asdetermined using the homographies described in greater detail above withrespect to FIGS. 2-7 ). For clarity and conciseness, the example of FIG.27 is described below in terms of global views of the space 102 (i.e., aview corresponding to the physical coordinates of the space 102). Thetracked object regions 2704, 2708, 2712 correspond to regions of thespace 102 associated with the positions of corresponding people (e.g.,or any other target object) moving through the space 102. For example,each tracked object region 2704, 2708, 2712 may correspond to adifferent person moving about in the space 102. Examples of determiningthe regions 2704, 2708, 2712 are described above, for example, withrespect to FIGS. 21, 22, and 24 . As one example, the tracked objectregions 2704, 2708, 2712 may be bounding boxes identified forcorresponding objects in the space 102. As another example, trackedobject regions 2704, 2708, 2712 may correspond to pixel masks determinedfor contours associated with the corresponding objects in the space 102(see, e.g., step 2104 of FIG. 21 for a more detailed description of thedetermination of a pixel mask). Generally, people may be tracked in thespace 102 and regions 2704, 2708, 2712 may be determined using anyappropriate tracking and identification method.

View 2702 at initial time t₀ includes a first tracked object region2704, a second tracked object region 2708, and a third tracked objectregion 2712. The view 2702 may correspond to a representation of thespace 102 from a top view with only the tracked object regions 2704,2708, 2712 shown (i.e., with other objects in the space 102 omitted). Attime t₀, the identities of all of the people are generally known (e.g.,because the people have recently entered the space 102 and/or becausethe people have not yet been near each other). The first tracked objectregion 2704 is associated with a first candidate list 2706, whichincludes a probability (P_(A)=100%) that the region 2704 (or thecorresponding person being tracked) is associated with a firstidentifier 2701 a. The second tracked object region 2708 is associatedwith a second candidate list 2710, which includes a probability(P_(B)=100%) that the region 2708 (or the corresponding person beingtracked) is associated with a second identifier 2701 b. The thirdtracked object region 2712 is associated with a third candidate list2714, which includes a probability (P_(C)=100%) that the region 2712 (orthe corresponding person being tracked) is associated with a thirdidentifier 2701 c. Accordingly, at time t₁, the candidate lists 2706,2710, 2714 indicate that the identity of each of the tracked objectregions 2704, 2708, 2712 is known with all probabilities having a valueof one hundred percent.

View 2716 shows positions of the tracked objects 2704, 2708, 2712 at afirst time t₁, which is after the initial time t₀. At time t₁, thetracking system detects an event which may cause the identities of thetracked object regions 2704, 2708 to be less certain. In this example,the tracking system 100 detects that the distance 2718 a between thefirst object region 274 and the second object region 2708 is less thanor equal to a threshold distance 2718 b. Because the tracked objectregions were near each other (i.e., within the threshold distance 2718b), there is a non-zero probability that the regions may bemisidentified during subsequent times. The threshold distance 2718 b maybe any appropriate distance, as described above with respect to FIG. 22. For example, the tracking system 100 may determine that the firstobject region 2704 is within the threshold distance 2718 b of the secondobject region 2708 by determining first coordinates of the first objectregion 2704, determining second coordinates of the second object region2708, calculating a distance 2718 a, and comparing distance 2718 a tothe threshold distance 2718 b. In some embodiments, the first and secondcoordinates correspond to pixel coordinates in an image capturing thefirst and second people, and the distance 2718 a corresponds to a numberof pixels between these pixel coordinates. For example, as illustratedin view 2716 of FIG. 27 , the distance 2718 a may correspond to thepixel distance between centroids of the tracked object regions 2704,2708. In other embodiments, the first and second coordinates correspondto physical, or global, coordinates in the space 102, and the distance2718 a corresponds to a physical distance (e.g., in units of length,such as inches). For example, physical coordinates may be determinedusing the homographies described in greater detail above with respect toFIGS. 2-7 .

After detecting that the identities of regions 2704, 2708 are lesscertain (i.e., that the first object region 2704 is within the thresholddistance 2718 b of the second object region 2708), the tracking system100 determines a probability 2717 that the first tracked object region2704 switched identifiers 2701 a-c with the second tracked object region2708. For example, when two contours become close in an image, there isa chance that the identities of the contours may be incorrect duringsubsequent tracking (e.g., because the tracking system 100 may assignthe wrong identifier 2701 a-c to the contours between frames). Theprobability 2717 that the identifiers 2701 a-c switched may bedetermined, for example, by accessing a predefined probability value(e.g., of 50%). In other cases, the probability 2717 may be based on thedistance 2718 a between the object regions 2704, 2708. For example, asthe distance 2718 decreases, the probability 2717 that the identifiers2701 a-c switched may increase. In the example of FIG. 27 , thedetermined probability 2717 is 20%, because the object regions 2704,2708 are relatively far apart but there is some overlap between theregions 2704, 2708.

In some embodiments, the tracking system 100 may determine a relativeorientation between the first object region 2704 and the second objectregion 2708, and the probability 2717 that the object regions 2704, 2708switched identifiers 2701 a-c may be based on this relative orientation.The relative orientation may correspond to an angle between a directiona person associated with the first region 2704 is facing and a directiona person associated with the second region 2708 is facing. For example,if the angle between the directions faced by people associated withfirst and second regions 2704, 2708 is near 180° (i.e., such that thepeople are facing in opposite directions), the probability 2717 thatidentifiers 2701 a-c switched may be decreased because this case maycorrespond to one person accidentally backing into the other person.

Based on the determined probability 2717 that the tracked object regions2704, 2708 switched identifiers 2701 a-c (e.g., 20% in this example),the tracking system 100 updates the first candidate list 2706 for thefirst object region 2704. The updated first candidate list 2706 includesa probability (P_(A)=80%) that the first region 2704 is associated withthe first identifier 2701 a and a probability (P_(B)=20%) that the firstregion 2704 is associated with the second identifier 2701 b. The secondcandidate list 2710 for the second object region 2708 is similarlyupdated based on the probability 2717 that the first object region 2704switched identifiers 2701 a-c with the second object region 2708. Theupdated second candidate list 2710 includes a probability (P_(A)=20%)that the second region 2708 is associated with the first identifier 2701a and a probability (P_(B)=80%) that the second region 2708 isassociated with the second identifier 2701 b.

View 2720 shows the object regions 2704, 2708, 2712 at a second timepoint t₂, which follows time t₁. At time t₂, a first personcorresponding to the first tracked region 2704 stands close to a thirdperson corresponding to the third tracked region 2712. In this examplecase, the tracking system 100 detects that the distance 2722 between thefirst object region 2704 and the third object region 2712 is less thanor equal to the threshold distance 2718 b (i.e., the same thresholddistance 2718 b described above with respect to view 2716). Afterdetecting that the first object region 2704 is within the thresholddistance 2718 b of the third object region 2712, the tracking system 100determines a probability 2721 that the first tracked object region 2704switched identifiers 2701 a-c with the third tracked object region 2712.As described above, the probability 2721 that the identifiers 2701 a-cswitched may be determined, for example, by accessing a predefinedprobability value (e.g., of 50%). In some cases, the probability 2721may be based on the distance 2722 between the object regions 2704, 2712.For example, since the distance 2722 is greater than distance 2718 a(from view 2716, described above), the probability 2721 that theidentifiers 2701 a-c switched may be greater at time t₁ than at time t₂.In the example of view 2720 of FIG. 27 , the determined probability 2721is 10% (which is smaller than the switching probability 2717 of 20%determined at time t₁).

Based on the determined probability 2721 that the tracked object regions2704, 2712 switched identifiers 2701 a-c (e.g., of 10% in this example),the tracking system 100 updates the first candidate list 2706 for thefirst object region 2704. The updated first candidate list 2706 includesa probability (P_(A)=73%) that the first object region 2704 isassociated with the first identifier 2701 a, a probability (P_(B)=17%)that the first object region 2704 is associated with the secondidentifier 2701 b, and a probability (P_(C)=10%) that the first objectregion 2704 is associated with the third identifier 2701 c. The thirdcandidate list 2714 for the third object region 2712 is similarlyupdated based on the probability 2721 that the first object region 2704switched identifiers 2701 a-c with the third object region 2712. Theupdated third candidate list 2714 includes a probability (P_(A)=7%) thatthe third object region 2712 is associated with the first identifier2701 a, a probability (P_(B)=3%) that the third object region 2712 isassociated with the second identifier 2701 b, and a probability(P_(C)=90%) that the third object region 2712 is associated with thethird identifier 2701 c. Accordingly, even though the third objectregion 2712 never interacted with (e.g., came within the thresholddistance 2718 b of) the second object region 2708, there is still anon-zero probability (P_(B)=3%) that the third object region 2712 isassociated with the second identifier 2701 b, which was originallyassigned (at time t₀) to the second object region 2708. In other words,the uncertainty in object identity that was detected at time t₁ ispropagated to the third object region 2712 via the interaction withregion 2704 at time t₂. This unique “propagation effect” facilitatesimproved object identification and can be used to narrow the searchspace (e.g., the number of possible identifiers 2701 a-c that may beassociated with a tracked object region 2704, 2708, 2712) when objectre-identification is needed (as described in greater detail below andwith respect to FIGS. 29-32 ).

View 2724 shows third object region 2712 and an unidentified objectregion 2726 at a third time point t₃, which follows time t₂. At time t₃,the first and second people associated with regions 2704, 2708 come intocontact (e.g., or “collide”) or are otherwise so close to one anotherthat the tracking system 100 cannot distinguish between the people. Forexample, contours detected for determining the first object region 2704and the second object region 2708 may have merged resulting in thesingle unidentified object region 2726. Accordingly, the position ofobject region 2726 may correspond to the position of one or both ofobject regions 2704 and 2708. At time t₃, the tracking system 100 maydetermine that the first and second object regions 2704, 2708 are nolonger detected because a first contour associated with the first objectregion 2704 is merged with a second contour associated with the secondobject region 2708.

The tracking system 100 may wait until a subsequent time t₄ (shown inview 2728) when the first and second object regions 2704, 2708 are againdetected before the candidate lists 2706, 2710 are updated. Time t₄generally corresponds to a time when the first and second peopleassociated with regions 2704, 2708 have separated from each other suchthat each person can be tracked in the space 102. Following a mergingevent such as is illustrated in view 2724, the probability 2725 thatregions 2704 and 2708 have switched identifiers 2701 a-c may be 50%. Attime t₄, updated candidate list 2706 includes an updated probability(P_(A)=60%) that the first object region 2704 is associated with thefirst identifier 2701 a, an updated probability (P_(B)=35%) that thefirst object region 2704 is associated with the second identifier 2701b, and an updated probability (P_(C)=5%) that the first object region2704 is associated with the third identifier 2701 c. Updated candidatelist 2710 includes an updated probability (P_(A)=33%) that the secondobject region 2708 is associated with the first identifier 2701 a, anupdated probability (P_(B)=62%) that the second object region 2708 isassociated with the second identifier 2701 b, and an updated probability(P_(C)=5%) that the second object region 2708 is associated with thethird identifier 2701 c. Candidate list 2714 is unchanged.

Still referring to view 2728, the tracking system 100 may determine thata highest value probability of a candidate list is less than a thresholdvalue (e.g., P_(threshold)=70%). In response to determining that thehighest probability of the first candidate list 2706 is less than thethreshold value, the corresponding object region 2704 may bere-identified (e.g., using any method of re-identification described inthis disclosure, for example, with respect to FIGS. 29-32 ). Forinstance, the first object region 2704 may be re-identified because thehighest probability (P_(A)=60%) is less than the threshold probability(P_(threshold)=70%). The tracking system 100 may extract features, ordescriptors, associated with observable characteristics of the firstperson (or corresponding contour) associated with the first objectregion 2704. The observable characteristics may be a height of theobject (e.g., determined from depth data received from a sensor), acolor associated with an area inside the contour (e.g., based on colorimage data from a sensor 108), a width of the object, an aspect ratio(e.g., width/length) of the object, a volume of the object (e.g., basedon depth data from sensor 108), or the like. Examples of otherdescriptors are described in greater detail below with respect to FIG.30 . As described in greater detail below, a texture feature (e.g.,determined using a local binary pattern histogram (LBPH) algorithm) maybe calculated for the person. Alternatively or additionally, anartificial neural network may be used to associate the person with thecorrect identifier 2701 a-c (e.g., as described in greater detail belowwith respect to FIG. 29-32 ).

Using the candidate lists 2706, 2710, 2714 may facilitate more efficientre-identification than was previously possible because, rather thanchecking all possible identifiers 2701 a-c (e.g., and other identifiersof people in space 102 not illustrated in FIG. 27 ) for a region 2704,2708, 2712 that has an uncertain identity, the tracking system 100 mayidentify a subset of all the other identifiers 2701 a-c that are mostlikely to be associated with the unknown region 2704, 2708, 2712 andonly compare descriptors of the unknown region 2704, 2708, 2712 todescriptors associated with the subset of identifiers 2701 a-c. In otherwords, if the identity of a tracked person is not certain, the trackingsystem 100 may only check to see if the person is one of the few peopleindicated in the person's candidate list, rather than comparing theunknown person to all of the people in the space 102. For example, onlyidentifiers 2701 a-c associated with a non-zero probability, or aprobability greater than a threshold value, in the candidate list 2706are likely to be associated with the correct identifier 2701 a-c of thefirst region 2704. In some embodiments, the subset may includeidentifiers 2701 a-c from the first candidate list 2706 withprobabilities that are greater than a threshold probability value (e.g.,of 10%). Thus, the tracking system 100 may compare descriptors of theperson associated with region 2704 to predetermined descriptorsassociated with the subset. As described in greater detail below withrespect to FIGS. 29-32 , the predetermined features (or descriptors) maybe determined when a person enters the space 102 and associated with theknown identifier 2701 a-c of the person during the entrance time period(i.e., before any events may cause the identity of the person to beuncertain. In the example of FIG. 27 , the object region 2708 may alsobe re-identified at or after time t₄ because the highest probabilityP_(B)=62% is less than the example threshold probability of 70%.

View 2730 corresponds to a time t₅ at which only the person associatedwith object region 2712 remains within the space 102. View 2730illustrates how the candidate lists 2706, 2710, 2714 can be used toensure that people only receive an exit notification 2734 when thesystem 100 is certain the person has exited the space 102. In theseembodiments, the tracking system 100 may be configured to transmit anexit notification 2734 to devices associated with these people when theprobability that a person has exited the space 102 is greater than anexit threshold (e.g., P_(exit)=95% or greater).

An exit notification 2734 is generally sent to the device of a personand includes an acknowledgement that the tracking system 100 hasdetermined that the person has exited the space 102. For example, if thespace 102 is a store, the exit notification 2734 provides a confirmationto the person that the tracking system 100 knows the person has exitedthe store and is, thus, no longer shopping. This may provide assuranceto the person that the tracking system 100 is operating properly and isno longer assigning items to the person or incorrectly charging theperson for items that he/she did not intend to purchase.

As people exit the space 102, the tracking system 100 may maintain arecord 2732 of exit probabilities to determine when an exit notification2734 should be sent. In the example of FIG. 27 , at time t₅ (shown inview 2730), the record 2732 includes an exit probability(P_(A,exit)=93%) that a first person associated with the first objectregion 2704 has exited the space 102. Since P_(A,exit) is less than theexample threshold exit probability of 95%, an exit notification 2734would not be sent to the first person (e.g., to his/her device). Thus,even though the first object region 2704 is no longer detected in thespace 102, an exit notification 2734 is not sent, because there is stilla chance that the first person is still in the space 102 (i.e., becauseof identity uncertainties that are captured and recorded via thecandidate lists 2706, 2710, 2714). This prevents a person from receivingan exit notification 2734 before he/she has exited the space 102. Therecord 2732 includes an exit probability (P_(B,exit)=97%) that thesecond person associated with the second object region 2708 has exitedthe space 102. Since P_(B,exit) is greater than the threshold exitprobability of 95%, an exit notification 2734 is sent to the secondperson (e.g., to his/her device). The record 2732 also includes an exitprobability (P_(C,exit)=10%) that the third person associated with thethird object region 2712 has exited the space 102. Since P_(C,exit) isless than the threshold exit probability of 95%, an exit notification2734 is not sent to the third person (e.g., to his/her device).

FIG. 28 is a flowchart of a method 2800 for creating and/or maintainingcandidate lists 2706, 2710, 2714 by tracking system 100. Method 2800generally facilitates improved identification of tracked people (e.g.,or other target objects) by maintaining candidate lists 2706, 2710, 2714which, for a given tracked person, or corresponding tracked objectregion (e.g., region 2704, 2708, 2712), include possible identifiers2701 a-c for the object and a corresponding probability that eachidentifier 2701 a-c is correct for the person. By maintaining candidatelists 2706, 2710, 2714 for tracked people, the people may be moreeffectively and efficiently identified during tracking. For example,costly person re-identification (e.g., in terms of system resourcesexpended) may only be used when a candidate list indicates that aperson's identity is sufficiently uncertain.

Method 2800 may begin at step 2802 where image frames are received fromone or more sensors 108. At step 2804, the tracking system 100 uses thereceived frames to track objects in the space 102. In some embodiments,tracking is performed using one or more of the unique tools described inthis disclosure (e.g., with respect to FIGS. 24-26 ). However, ingeneral, any appropriate method of sensor-based object tracking may beemployed.

At step 2806, the tracking system 100 determines whether a first personis within a threshold distance 2718 b of a second person. This case maycorrespond to the conditions shown in view 2716 of FIG. 27 , describedabove, where first object region 2704 is distance 2718 a away fromsecond object region 2708. As described above, the distance 2718 a maycorrespond to a pixel distance measured in a frame or a physicaldistance in the space 102 (e.g., determined using a homographyassociating pixel coordinates to physical coordinates in the space 102).If the first and second people are not within the threshold distance2718 b of each other, the system 100 continues tracking objects in thespace 102 (i.e., by returning to step 2804).

However, if the first and second people are within the thresholddistance 2718 b of each other, method 2800 proceeds to step 2808, wherethe probability 2717 that the first and second people switchedidentifiers 2701 a-c is determined. As described above, the probability2717 that the identifiers 2701 a-c switched may be determined, forexample, by accessing a predefined probability value (e.g., of 50%). Insome embodiments, the probability 2717 is based on the distance 2718 abetween the people (or corresponding object regions 2704, 2708), asdescribed above. In some embodiments, as described above, the trackingsystem 100 determines a relative orientation between the first personand the second person, and the probability 2717 that the people (orcorresponding object regions 2704, 2708) switched identifiers 2701 a-cis determined, at least in part, based on this relative orientation.

At step 2810, the candidate lists 2706, 2710 for the first and secondpeople (or corresponding object regions 2704, 2708) are updated based onthe probability 2717 determined at step 2808. For instance, as describedabove, the updated first candidate list 2706 may include a probabilitythat the first object is associated with the first identifier 2701 a anda probability that the first object is associated with the secondidentifier 2701 b. The second candidate list 2710 for the second personis similarly updated based on the probability 2717 that the first objectswitched identifiers 2701 a-c with the second object (determined at step2808). The updated second candidate list 2710 may include a probabilitythat the second person is associated with the first identifier 2701 aand a probability that the second person is associated with the secondidentifier 2701 b.

At step 2812, the tracking system 100 determines whether the firstperson (or corresponding region 2704) is within a threshold distance2718 b of a third object (or corresponding region 2712). This case maycorrespond, for example, to the conditions shown in view 2720 of FIG. 27, described above, where first object region 2704 is distance 2722 awayfrom third object region 2712. As described above, the thresholddistance 2718 b may correspond to a pixel distance measured in a frameor a physical distance in the space 102 (e.g., determined using anappropriate homography associating pixel coordinates to physicalcoordinates in the space 102).

If the first and third people (or corresponding regions 2704 and 2712)are within the threshold distance 2718 b of each other, method 2800proceeds to step 2814, where the probability 2721 that the first andthird people (or corresponding regions 2704 and 2712) switchedidentifiers 2701 a-c is determined. As described above, this probability2721 that the identifiers 2701 a-c switched may be determined, forexample, by accessing a predefined probability value (e.g., of 50%). Theprobability 2721 may also or alternatively be based on the distance 2722between the objects 2727 and/or a relative orientation of the first andthird people, as described above. At step 2816, the candidate lists2706, 2714 for the first and third people (or corresponding regions2704, 2712) are updated based on the probability 2721 determined at step2808. For instance, as described above, the updated first candidate list2706 may include a probability that the first person is associated withthe first identifier 2701 a, a probability that the first person isassociated with the second identifier 2701 b, and a probability that thefirst object is associated with the third identifier 2701 c. The thirdcandidate list 2714 for the third person is similarly updated based onthe probability 2721 that the first person switched identifiers with thethird person (i.e., determined at step 2814). The updated thirdcandidate list 2714 may include, for example, a probability that thethird object is associated with the first identifier 2701 a, aprobability that the third object is associated with the secondidentifier 2701 b, and a probability that the third object is associatedwith the third identifier 2701 c. Accordingly, if the steps of method2800 proceed in the example order illustrated in FIG. 28 , the candidatelist 2714 of the third person includes a non-zero probability that thethird object is associated with the second identifier 2701 b, which wasoriginally associated with the second person.

If, at step 2812, the first and third people (or corresponding regions2704 and 2712) are not within the threshold distance 2718 b of eachother, the system 100 generally continues tracking people in the space102. For example, the system 100 may proceed to step 2818 to determinewhether the first person is within a threshold distance of an n^(th)person (i.e., some other person in the space 102). At step 2820, thesystem 100 determines the probability that the first and n^(th) peopleswitched identifiers 2701 a-c, as described above, for example, withrespect to steps 2808 and 2814. At step 2822, the candidate lists forthe first and n^(th) people are updated based on the probabilitydetermined at step 2820, as described above, for example, with respectto steps 2810 and 2816 before method 2800 ends. If, at step 2818, thefirst person is not within the threshold distance of the n^(th) person,the method 2800 proceeds to step 2824.

At step 2824, the tracking system 100 determines if a person has exitedthe space 102. For instance, as described above, the tracking system 100may determine that a contour associated with a tracked person is nolonger detected for at least a threshold time period (e.g., of about 30seconds or more). The system 100 may additionally determine that aperson exited the space 102 when a person is no longer detected and alast determined position of the person was at or near an exit position(e.g., near a door leading to a known exit from the space 102). If aperson has not exited the space 102, the tracking system 100 continuesto track people (e.g., by returning to step 2802).

If a person has exited the space 102, the tracking system 100 calculatesor updates record 2732 of probabilities that the tracked objects haveexited the space 102 at step 2826. As described above, each exitprobability of record 2732 generally corresponds to a probability that aperson associated with each identifier 2701 a-c has exited the space102. At step 2828, the tracking system 100 determines if a combined exitprobability in the record 2732 is greater than a threshold value (e.g.,of 95% or greater). If a combined exit probability is not greater thanthe threshold, the tracking system 100 continues to track objects (e.g.,by continuing to step 2818).

If an exit probability from record 2732 is greater than the threshold, acorresponding exit notification 2734 may be sent to the person linked tothe identifier 2701 a-c associated with the probability at step 2830, asdescribed above with respect to view 2730 of FIG. 27 . This may preventor reduce instances where an exit notification 2734 is sent prematurelywhile an object is still in the space 102. For example, it may bebeneficial to delay sending an exit notification 2734 until there is ahigh certainty that the associated person is no longer in the space 102.In some cases, several tracked people must exit the space 102 before anexit probability in record 2732 for a given identifier 2701 a-c issufficiently large for an exit notification 2734 to be sent to theperson (e.g., to a device associated with the person).

Modifications, additions, or omissions may be made to method 2800depicted in FIG. 28 . Method 2800 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 100 orcomponents thereof performing steps, any suitable system or componentsof the system may perform one or more steps of the method 2800.

Person Re-Identification

As described above, in some cases, the identity of a tracked person canbecome unknown (e.g., when the people become closely spaced or“collide”, or when the candidate list of a person indicates the person'sidentity is not known, as described above with respect to FIGS. 27-28 ),and the person may need to be re-identified. This disclosurecontemplates a unique approach to efficiently and reliablyre-identifying people by the tracking system 100. For example, ratherthan relying entirely on resource-expensive machine learning-basedapproaches to re-identify people, a more efficient and speciallystructured approach may be used where “lower-cost” descriptors relatedto observable characteristics (e.g., height, color, width, volume, etc.)of people are used first for person re-identification. “Higher-cost”descriptors (e.g., determined using artificial neural network models)are only used when the lower-cost methods cannot provide reliableresults. For instance, in some embodiments, a person may first bere-identified based on his/her height, hair color, and/or shoe color.However, if these descriptors are not sufficient for reliablyre-identifying the person (e.g., because other people being tracked havesimilar characteristics), progressively higher-level approaches may beused (e.g., involving artificial neural networks that are trained torecognize people) which may be more effective at person identificationbut which generally involve the use of more processing resources.

As an example, each person's height may be used initially forre-identification. However, if another person in the space 102 has asimilar height, a height descriptor may not be sufficient forre-identifying the people (e.g., because it is not possible todistinguish between people with similar heights based on height alone),and a higher-level approach may be used (e.g., using a texture operatoror an artificial neural network to characterize the person). In someembodiments, if the other person with a similar height has neverinteracted with the person being re-identified (e.g., as recorded ineach person's candidate list—see FIG. 27 and corresponding descriptionabove), height may still be an appropriate feature for re-identifyingthe person (e.g., because the other person with a similar height is notassociated with a candidate identity of the person being re-identified).

FIG. 29 illustrates a tracking subsystem 2900 configured to track people(e.g., and/or other target objects) based on sensor data 2904 receivedfrom one or more sensors 108. In general, the tracking subsystem 2900may include one or both of the server 106 and the client(s) 105 of FIG.1 , described above. Tracking subsystem 2900 may be implemented usingthe device 3800 described below with respect to FIG. 38 . Trackingsubsystem 2900 may track object positions 2902, over a period of timeusing sensor data 2904 (e.g., top-view images) generated by at least oneof sensors 108. Object positions 2902 may correspond to local pixelpositions (e.g., pixel positions 2226, 2234 of FIG. 22 ) determined at asingle sensor 108 and/or global positions corresponding to physicalpositions (e.g., positions 2228 of FIG. 22 ) in the space 102 (e.g.,using the homographies described above with respect to FIGS. 2-7 ). Insome cases, object positions 2902 may correspond to regions detected inan image, or in the space 102, that are associated with the location ofa corresponding person (e.g., regions 2704, 2708, 2712 of FIG. 27 ,described above). People may be tracked and corresponding positions 2902may be determined, for example, based on pixel coordinates of contoursdetected in top-view images generated by sensor(s) 108. Examples ofcontour-based detection and tracking are described above, for example,with respect to FIGS. 24 and 27 . However, in general, any appropriatemethod of sensor-based tracking may be used to determine positions 2902.

For each object position 2902, the subsystem 2900 maintains acorresponding candidate list 2906 (e.g., as described above with respectto FIG. 27 ). The candidate lists 2906 are generally used to maintain arecord of the most likely identities of each person being tracked (i.e.,associated with positions 2902). Each candidate list 2906 includesprobabilities which are associated with identifiers 2908 of people thathave entered the space 102. The identifiers 2908 may be any appropriaterepresentation (e.g., an alphanumeric string, or the like) foridentifying a person (e.g., a username, name, account number, or thelike associated with the person being tracked). In some embodiments, theidentifiers 2908 may be anonymized (e.g., using hashing or any otherappropriate anonymization technique).

Each of the identifiers 2908 is associated with one or morepredetermined descriptors 2910. The predetermined descriptors 2910generally correspond to information about the tracked people that can beused to re-identify the people when necessary (e.g., based on thecandidate lists 2906). The predetermined descriptors 2910 may includevalues associated with observable and/or calculated characteristics ofthe people associated with the identifiers 2908. For instance, thedescriptors 2910 may include heights, hair colors, clothing colors, andthe like. As described in greater detail below, the predetermineddescriptors 2910 are generally determined by the tracking subsystem 2900during an initial time period (e.g., when a person associated with agiven tracked position 2902 enters the space) and are used tore-identify people associated with tracked positions 2902 when necessary(e.g., based on candidate lists 2906).

When re-identification is needed (or periodically during tracking) for agiven person at position 2902, the tracking subsystem 2900 may determinemeasured descriptors 2912 for the person associated with the position2902. FIG. 30 illustrates the determination of descriptors 2910, 2912based on a top-view depth image 3002 received from a sensor 108. Arepresentation 2904 a of a person corresponding to the tracked objectposition 2902 is observable in the image 3002. The tracking subsystem2900 may detect a contour 3004 b associated with the representation 3004a. The contour 3004 b may correspond to a boundary of the representation3004 a (e.g., determined at a given depth in image 3002). Trackingsubsystem 2900 generally determines descriptors 2910, 2912 based on therepresentation 3004 a and/or the contour 3004 b. In some cases, therepresentation 3004 b appears within a predefined region-of-interest3006 of the image 3002 in order for descriptors 2910, 2912 to bedetermined by the tracking subsystem 2900. This may facilitate morereliable descriptor 2910, 2912 determination, for example, becausedescriptors 2910, 2912 may be more reproducible and/or reliable when theperson being imaged is located in the portion of the sensor'sfield-of-view that corresponds to this region-of-interest 3006. Forexample, descriptors 2910, 2912 may have more consistent values when theperson is imaged within the region-of-interest 3006.

Descriptors 2910, 2912 determined in this manner may include, forexample, observable descriptors 3008 and calculated descriptors 3010.For example, the observable descriptors 3008 may correspond tocharacteristics of the representation 3004 a and/or contour 3004 b whichcan be extracted from the image 3002 and which correspond to observablefeatures of the person. Examples of observable descriptors 3008 includea height descriptor 3012 (e.g., a measure of the height in pixels orunits of length) of the person based on representation 3004 a and/orcontour 3004 b), a shape descriptor 3014 (e.g., width, length, aspectratio, etc.) of the representation 3004 a and/or contour 3004 b, avolume descriptor 3016 of the representation 3004 a and/or contour 3004b, a color descriptor 3018 of representation 3004 a (e.g., a color ofthe person's hair, clothing, shoes, etc.), an attribute descriptor 3020associated with the appearance of the representation 3004 a and/orcontour 3004 b (e.g., an attribute such as “wearing a hat,” “carrying achild,” “pushing a stroller or cart,”), and the like.

In contrast to the observable descriptors 3008, the calculateddescriptors 3010 generally include values (e.g., scalar or vectorvalues) which are calculated using the representation 3004 a and/orcontour 3004 b and which do not necessarily correspond to an observablecharacteristic of the person. For example, the calculated descriptors3010 may include image-based descriptors 3022 and model-baseddescriptors 3024. Image-based descriptors 3022 may, for example, includeany descriptor values (i.e., scalar and/or vector values) calculatedfrom image 3002. For example, a texture operator such as a local binarypattern histogram (LBPH) algorithm may be used to calculate a vectorassociated with the representation 3004 a. This vector may be stored asa predetermined descriptor 2910 and measured at subsequent times as adescriptor 2912 for re-identification. Since the output of a textureoperator, such as the LBPH algorithm may be large (i.e., in terms of theamount of memory required to store the output), it may be beneficial toselect a subset of the output that is most useful for distinguishingpeople. Accordingly, in some cases, the tracking subsystem 2900 mayselect a portion of the initial data vector to include in the descriptor2910, 2912. For example, a principal component analysis may be used toselect and retain a portion of the initial data vector that is mostuseful for effective person re-identification.

In contrast to the image-based descriptors 3022, model-based descriptors3024 are generally determined using a predefined model, such as anartificial neural network. For example, a model-based descriptor 3024may be the output (e.g., a scalar value or vector) output by anartificial neural network trained to recognize people based on theircorresponding representation 3004 a and/or contour 3004 b in top-viewimage 3002. For example, a Siamese neural network may be trained toassociate representations 3004 a and/or contours 3004 b in top-viewimages 3002 with corresponding identifiers 2908 and subsequentlyemployed for re-identification 2929.

Returning to FIG. 29 , the descriptor comparator 2914 of the trackingsubsystem 2900 may be used to compare the measured descriptor 2912 tocorresponding predetermined descriptors 2910 in order to determine thecorrect identity of a person being tracked. For example, the measureddescriptor 2912 may be compared to a corresponding predetermineddescriptor 2910 in order to determine the correct identifier 2908 forthe person at position 2902. For instance, if the measured descriptor2912 is a height descriptor 3012, it may be compared to predeterminedheight descriptors 2910 for identifiers 2908, or a subset of theidentifiers 2908 determined using the candidate list 2906. Comparing thedescriptors 2910, 2912 may involve calculating a difference betweenscalar descriptor values (e.g., a difference in heights 3012, volumes3018, etc.), determining whether a value of a measured descriptor 2912is within a threshold range of the corresponding predetermineddescriptor 2910 (e.g., determining if a color value 3018 of the measureddescriptor 2912 is within a threshold range of the color value 3018 ofthe predetermined descriptor 2910), determining a cosine similarityvalue between vectors of the measured descriptor 2912 and thecorresponding predetermined descriptor 2910 (e.g., determining a cosinesimilarity value between a measured vector calculated using a textureoperator or neural network and a predetermined vector calculated in thesame manner). In some embodiments, only a subset of the predetermineddescriptors 2910 are compared to the measured descriptor 2912. Thesubset may be selected using the candidate list 2906 for the person atposition 2902 that is being re-identified. For example, the person'scandidate list 2906 may indicate that only a subset (e.g., two, three,or so) of a larger number of identifiers 2908 are likely to beassociated with the tracked object position 2902 that requiresre-identification.

When the correct identifier 2908 is determined by the descriptorcomparator 2914, the comparator 2914 may update the candidate list 2906for the person being re-identified at position 2902 (e.g., by sendingupdate 2916). In some cases, a descriptor 2912 may be measured for anobject that does not require re-identification (e.g., a person for whichthe candidate list 2906 indicates there is 100% probability that theperson corresponds to a single identifier 2908). In these cases,measured identifiers 2912 may be used to update and/or maintain thepredetermined descriptors 2910 for the person's known identifier 2908(e.g., by sending update 2918). For instance, a predetermined descriptor2910 may need to be updated if a person associated with the position2902 has a change of appearance while moving through the space 102(e.g., by adding or removing an article of clothing, by assuming adifferent posture, etc.).

FIG. 31A illustrates positions over a period of time of tracked people3102, 3104, 3106, during an example operation of tracking system 2900.The first person 3102 has a corresponding trajectory 3108 represented bythe solid line in FIG. 31A. Trajectory 3108 corresponds to the historyof positions of person 3102 in the space 102 during the period of time.Similarly, the second person 3104 has a corresponding trajectory 3110represented by the dashed-dotted line in FIG. 31A. Trajectory 3110corresponds to the history of positions of person 3104 in the space 102during the period of time. The third person 3106 has a correspondingtrajectory 3112 represented by the dotted line in FIG. 31A. Trajectory3112 corresponds to the history of positions of person 3112 in the space102 during the period of time.

When each of the people 3102, 3104, 3106 first enter the space 102(e.g., when they are within region 3114), predetermined descriptors 2910are generally determined for the people 3102, 3104, 3106 and associatedwith the identifiers 2908 of the people 3102, 3104, 3106. Thepredetermined descriptors 2910 are generally accessed when the identityof one or more of the people 3102, 3104, 3106 is not sufficientlycertain (e.g., based on the corresponding candidate list 2906 and/or inresponse to a “collision event,” as described below) in order tore-identify the person 3102, 3104, 3106. For example, re-identificationmay be needed following a “collision event” between two or more of thepeople 3102, 3104, 3106. A collision event typically corresponds to animage frame in which contours associated with different people merge toform a single contour (e.g., the detection of merged contour 2220 shownin FIG. 22 may correspond to detecting a collision event). In someembodiments, a collision event corresponds to a person being locatedwithin a threshold distance of another person (see, e.g., distance 2718a and 2722 in FIG. 27 and the corresponding description above). Moregenerally, a collision event may correspond to any event that results ina person's candidate list 2906 indicating that re-identification isneeded (e.g., based on probabilities stored in the candidate list2906—see FIGS. 27-28 and the corresponding description above).

In the example of FIG. 31A, when the people 3102, 3104, 3106 are withinregion 3114, the tracking subsystem 2900 may determine a first heightdescriptor 3012 associated with a first height of the first person 3102,a first contour descriptor 3014 associated with a shape of the firstperson 3102, a first anchor descriptor 3024 corresponding to a firstvector generated by an artificial neural network for the first person3102, and/or any other descriptors 2910 described with respect to FIG.30 above. Each of these descriptors is stored for use as a predetermineddescriptor 2910 for re-identifying the first person 3102. Thesepredetermined descriptors 2910 are associated with the first identifier(i.e., of identifiers 2908) of the first person 3102. When the identityof the first person 3102 is certain (e.g., prior to the first collisionevent at position 3116), each of the descriptors 2910 described abovemay be determined again to update the predetermined descriptors 2910.For example, if person 3102 moves to a position in the space 102 thatallows the person 3102 to be within a desired region-of-interest (e.g.,region-of-interest 3006 of FIG. 30 ), new descriptors 2912 may bedetermined. The tracking subsystem 2900 may use these new descriptors2912 to update the previously determined descriptors 2910 (e.g., seeupdate 2918 of FIG. 29 ). By intermittently updating the predetermineddescriptors 2910, changes in the appearance of people being tracked canbe accounted for (e.g., if a person puts on or removes an article ofclothing, assumes a different posture, etc.).

At a first timestamp associated with a time t₁, the tracking subsystem2900 detects a collision event between the first person 3102 and thirdperson 3106 at position 3116 illustrated in FIG. 31A. For example, thecollision event may correspond to a first tracked position of the firstperson 3102 being within a threshold distance of a second trackedposition of the third person 3106 at the first timestamp. In someembodiments, the collision event corresponds to a first contourassociated with the first person 3102 merging with a third contourassociated with the third person 3106 at the first timestamp. Moregenerally, the collision event may be associated with any occurrencewhich causes a highest value probability of a candidate list associatedwith the first person 3102 and/or the third person 3106 to fall below athreshold value (e.g., as described above with respect to view 2728 ofFIG. 27 ). In other words, any event causing the identity of person 3102to become uncertain may be considered a collision event.

After the collision event is detected, the tracking subsystem 2900receives a top-view image (e.g., top-view image 3002 of FIG. 30 ) fromsensor 108. The tracking subsystem 2900 determines, based on thetop-view image, a first descriptor for the first person 3102. Asdescribed above, the first descriptor includes at least one valueassociated with an observable, or calculated, characteristic of thefirst person 3104 (e.g., of representation 3004 a and/or contour 3004 bof FIG. 30 ). In some embodiments, the first descriptor may be a“lower-cost” descriptor that requires relatively few processingresources to determine, as described above. For example, the trackingsubsystem 2900 may be able to determine a lower-cost descriptor moreefficiently than it can determine a higher-cost descriptor (e.g., amodel-based descriptor 3024 described above with respect to FIG. 30 ).For instance, a first number of processing cores used to determine thefirst descriptor may be less than a second number of processing coresused to determine a model-based descriptor 3024 (e.g., using anartificial neural network). Thus, it may be beneficial to re-identify aperson, whenever possible, using a lower-cost descriptor wheneverpossible.

However, in some cases, the first descriptor may not be sufficient forre-identifying the first person 3102. For example, if the first person3102 and the third person 3106 correspond to people with similarheights, a height descriptor 3012 generally cannot be used todistinguish between the people 3102, 3106. Accordingly, before the firstdescriptor 2912 is used to re-identify the first person 3102, thetracking subsystem 2900 may determine whether certain criteria aresatisfied for distinguishing the first person 3102 from the third person3106 based on the first descriptor 2912. In some embodiments, thecriteria are not satisfied when a difference, determined during a timeinterval associated with the collision event (e.g., at a time at or neartime t₁), between the descriptor 2912 of the first person 3102 and acorresponding descriptor 2912 of the third person 3106 is less than aminimum value.

FIG. 31B illustrates the evaluation of these criteria based on thehistory of descriptor values for people 3102 and 3106 over time. Plot3150, shown in FIG. 31B, shows a first descriptor value 3152 for thefirst person 3102 over time and a second descriptor value 3154 for thethird person 3106 over time. In general, descriptor values may fluctuateover time because of changes in the environment, the orientation ofpeople relative to sensors 108, sensor variability, changes inappearance, etc. The descriptor values 3152, 3154 may be associated witha shape descriptor 3014, a volume 3016, a contour-based descriptor 3022,or the like, as described above with respect to FIG. 30 . At time t₁,the descriptor values 3152, 3154 have a relatively large difference 3156that is greater than the threshold difference 3160, illustrated in FIG.31B. Accordingly, in this example, at or near (e.g., within a brief timeinterval of a few seconds or minutes following t₁), the criteria aresatisfied and the descriptor 2912 associated with descriptor values3152, 3154 can generally be used to re-identify the first and thirdpeople 3102, 3106.

When the criteria are satisfied for distinguishing the first person 3102from the third person 3106 based on the first descriptor 2912 (as is thecase at t₁), the descriptor comparator 2914 may compare the firstdescriptor 2912 for the first person 3102 to each of the correspondingpredetermined descriptors 2910 (i.e., for all identifiers 2908).However, in some embodiments, comparator 2914 may compare the firstdescriptor 2912 for the first person 3102 to predetermined descriptors2910 for only a select subset of the identifiers 2908. The subset may beselected using the candidate list 2906 for the person that is beingre-identified (see, e.g., step 3208 of method 3200 described below withrespect to FIG. 32 ). For example, the person's candidate list 2906 mayindicate that only a subset (e.g., two, three, or so) of a larger numberof identifiers 2908 are likely to be associated with the tracked objectposition 2902 that requires re-identification. Based on this comparison,the tracking subsystem 2900 may identify the predetermined descriptor2910 that is most similar to the first descriptor 2912. For example, thetracking subsystem 2900 may determine that a first identifier 2908corresponds to the first person 3102 by, for each member of the set (orthe determined subset) of the predetermined descriptors 2910,calculating an absolute value of a difference in a value of the firstdescriptor 2912 and a value of the predetermined descriptor 2910. Thefirst identifier 2908 may be selected as the identifier 2908 associatedwith the smallest absolute value.

Referring again to FIG. 31A, at time t₂, a second collision event occursat position 3118 between people 3102, 3106. Turning back to FIG. 31B,the descriptor values 3152, 3154 have a relatively small difference 3158at time t₂ (e.g., compared to difference 3156 at time t₁), which is lessthan the threshold value 3160. Thus, at time t₂, the descriptor 2912associated with descriptor values 3152, 3154 generally cannot be used tore-identify the first and third people 3102, 3106, and the criteria forusing the first descriptor 2912 are not satisfied. Instead, a different,and likely a “higher-cost” descriptor 2912 (e.g., a model-baseddescriptor 3024) should be used to re-identify the first and thirdpeople 3102, 3106 at time t₂.

For example, when the criteria are not satisfied for distinguishing thefirst person 3102 from the third person 3106 based on the firstdescriptor 2912 (as is the case in this example at time t₂), thetracking subsystem 2900 determines a new descriptor 2912 for the firstperson 3102. The new descriptor 2912 is typically a value or vectorgenerated by an artificial neural network configured to identify peoplein top-view images (e.g., a model-based descriptor 3024 of FIG. 30 ).The tracking subsystem 2900 may determine, based on the new descriptor2912, that a first identifier 2908 from the predetermined identifiers2908 (or a subset determined based on the candidate list 2906, asdescribed above) corresponds to the first person 3102. For example, thetracking subsystem 2900 may determine that the first identifier 2908corresponds to the first person 3102 by, for each member of the set (orsubset) of predetermined identifiers 2908, calculating an absolute valueof a difference in a value of the first identifier 2908 and a value ofthe predetermined descriptors 2910. The first identifier 2908 may beselected as the identifier 2908 associated with the smallest absolutevalue.

In cases where the second descriptor 2912 cannot be used to reliablyre-identify the first person 3102 using the approach described above,the tracking subsystem 2900 may determine a measured descriptor 2912 forall of the “candidate identifiers” of the first person 3102. Thecandidate identifiers generally refer to the identifiers 2908 of people(e.g., or other tracked objects) that are known to be associated withidentifiers 2908 appearing in the candidate list 2906 of the firstperson 3102 (e.g., as described above with respect to FIGS. 27 and 28 ).For instance, the candidate identifiers may be identifiers 2908 oftracked people (i.e., at tracked object positions 2902) that appear inthe candidate list 2906 of the person being re-identified. FIG. 31Cillustrates how predetermined descriptors 3162, 3164, 3166 for a first,second, and third identifier 2908 may be compared to each of themeasured descriptors 3168, 3170, 3172 for people 3102, 3104, 3106. Thecomparison may involve calculating a cosine similarity value betweenvectors associated with the descriptors. Based on the results of thecomparison, each person 3102, 3104, 3106 is assigned the identifier 2908corresponding to the best-matching predetermined descriptor 3162, 3164,3166. A best matching descriptor may correspond to a highest cosinesimilarity value (i.e., nearest to one).

FIG. 32 illustrates a method 3200 for re-identifying tracked peopleusing tracking subsystem 2900 illustrated in FIG. 29 and describedabove. The method 3200 may begin at step 3202 where the trackingsubsystem 2900 receives top-view image frames from one or more sensors108. At step 3204, the tracking subsystem 2900 tracks a first person3102 and one or more other people (e.g., people 3104, 3106) in the space102 using at least a portion of the top-view images generated by thesensors 108. For instance, tracking may be performed as described abovewith respect to FIGS. 24-26 , or using any appropriate object trackingalgorithm. The tracking subsystem 2900 may periodically determineupdated predetermined descriptors associated with the identifiers 2908(e.g., as described with respect to update 2918 of FIG. 29 ). In someembodiments, the tracking subsystem 2900, in response to determining theupdated descriptors, determines that one or more of the updatedpredetermined descriptors is different by at least a threshold amountfrom a corresponding previously predetermined descriptor 2910. In thiscase, the tracking subsystem 2900 may save both the updated descriptorand the corresponding previously predetermined descriptor 2910. This mayallow for improved re-identification when characteristics of the peoplebeing tracked may change intermittently during tracking.

At step 3206, the tracking subsystem 2900 determines whetherre-identification of the first tracked person 3102 is needed. This maybe based on a determination that contours have merged in an image frame(e.g., as illustrated by merged contour 2220 of FIG. 22 ) or on adetermination that a first person 3102 and a second person 3104 arewithin a threshold distance (e.g., distance 2918 b of FIG. 29 ) of eachother, as described above. In some embodiments, a candidate list 2906may be used to determine that re-identification of the first person 3102is required. For instance, if a highest probability from the candidatelist 2906 associated with the tracked person 3102 is less than athreshold value (e.g., 70%), re-identification may be needed (see alsoFIGS. 27-28 and the corresponding description above). Ifre-identification is not needed, the tracking subsystem 2900 generallycontinues to track people in the space (e.g., by returning to step3204).

If the tracking subsystem 2900 determines at step 3206 thatre-identification of the first tracked person 3102 is needed, thetracking subsystem 2900 may determine candidate identifiers for thefirst tracked person 3102 at step 3208. The candidate identifiersgenerally include a subset of all of the identifiers 2908 associatedwith tracked people in the space 102, and the candidate identifiers maybe determined based on the candidate list 2906 for the first trackedperson 3102. In other words, the candidate identifiers are a subset ofthe identifiers 2906 which are most likely to include the correctidentifier 2908 for the first tracked person 3102 based on a history ofmovements of the first tracked person 3102 and interactions of the firsttracked person 3102 with the one or more other tracked people 3104, 3106in the space 102 (e.g., based on the candidate list 2906 that is updatedin response to these movements and interactions).

At step 3210, the tracking subsystem 2900 determines a first descriptor2912 for the first tracked person 3102. For example, the trackingsubsystem 2900 may receive, from a first sensor 108, a first top-viewimage of the first person 3102 (e.g., such as image 3002 of FIG. 30 ).For instance, as illustrated in the example of FIG. 30 , in someembodiments, the image 3002 used to determine the descriptor 2912includes the representation 3004 a of the object within aregion-of-interest 3006 within the full frame of the image 3002. Thismay provide for more reliable descriptor 2912 determination. In someembodiments, the image data 2904 include depth data (i.e., image data atdifferent depths). In such embodiments, the tracking subsystem 2900 maydetermine the descriptor 2912 based on a depth region-of-interest, wherethe depth region-of-interest corresponds to depths in the imageassociated with the head of person 3102. In these embodiments,descriptors 2912 may be determined that are associated withcharacteristics or features of the head of the person 3102.

At step 3212, the tracking subsystem 2900 may determine whether thefirst descriptor 2912 can be used to distinguish the first person 3102from the candidate identifiers (e.g., one or both of people 3104, 3106)by, for example, determining whether certain criteria are satisfied fordistinguishing the first person 3102 from the candidates based on thefirst descriptor 2912. In some embodiments, the criteria are notsatisfied when a difference, determined during a time intervalassociated with the collision event, between the first descriptor 2912and corresponding descriptors 2910 of the candidates is less than aminimum value, as described in greater detail above with respect toFIGS. 31A,B.

If the first descriptor can be used to distinguish the first person 3102from the candidates (e.g., as was the case at time t₁ in the example ofFIG. 31A,B), the method 3200 proceeds to step 3214 at which point thetracking subsystem 2900 determines an updated identifier for the firstperson 3102 based on the first descriptor 2912. For example, thetracking subsystem 2900 may compare (e.g., using comparator 2914) thefirst descriptor 2912 to the set of predetermined descriptors 2910 thatare associated with the candidate objects determined for the firstperson 3102 at step 3208. In some embodiments, the first descriptor 2912is a data vector associated with characteristics of the first person inthe image (e.g., a vector determined using a texture operator such asthe LBPH algorithm), and each of the predetermined descriptors 2910includes a corresponding predetermined data vector (e.g., determined foreach tracked pers 3102, 3104, 3106 upon entering the space 102). In suchembodiments, the tracking subsystem 2900 compares the first descriptor2912 to each of the predetermined descriptors 2910 associated with thecandidate objects by calculating a cosine similarity value between thefirst data vector and each of the predetermined data vectors. Thetracking subsystem 2900 determines the updated identifier as theidentifier 2908 of the candidate object with the cosine similarity valuenearest one (i.e., the vector that is most “similar” to the vector ofthe first descriptor 2912).

At step 3216, the identifiers 2908 of the other tracked people 3104,3106 may be updated as appropriate by updating other people's candidatelists 2906. For example, if the first tracked person 3102 was found tobe associated with an identifier 2908 that was previously associatedwith the second tracked person 3104. Steps 3208 to 3214 may be repeatedfor the second person 3104 to determine the correct identifier 2908 forthe second person 3104. In some embodiments, when the identifier 2908for the first person 3102 is updated, the identifiers 2908 for people(e.g., one or both of people 3104 and 3106) that are associated with thefirst person's candidate list 2906 are also updated at step 3216. As anexample, the candidate list 2906 of the first person 3102 may have anon-zero probability that the first person 3102 is associated with asecond identifier 2908 originally linked to the second person 3104 and athird probability that the first person 3102 is associated with a thirdidentifier 2908 originally linked to the third person 3106. In thiscase, after the identifier 2908 of the first person 3102 is updated, theidentifiers 2908 of the second and third people 3104, 3106 may also beupdated according to steps 3208-3214.

If, at step 3212, the first descriptor 2912 cannot be used todistinguish the first person 3102 from the candidates (e.g., as was thecase at time t₂ in the example of FIG. 31A,B), the method 3200 proceedsto step 3218 to determine a second descriptor 2912 for the first person3102. As described above, the second descriptor 2912 may be a“higher-level” descriptor such as a model-based descriptor 3024 of FIG.30 ). For example, the second descriptor 2912 may be less efficient(e.g., in terms of processing resources required) to determine than thefirst descriptor 2912. However, the second descriptor 2912 may be moreeffective and reliable, in some cases, for distinguishing betweentracked people.

At step 3220, the tracking system 2900 determines whether the seconddescriptor 2912 can be used to distinguish the first person 3102 fromthe candidates (from step 3218) using the same or a similar approach tothat described above with respect to step 3212. For example, thetracking subsystem 2900 may determine if the cosine similarity valuesbetween the second descriptor 2912 and the predetermined descriptors2910 are greater than a threshold cosine similarity value (e.g., of0.5). If the cosine similarity value is greater than the threshold, thesecond descriptor 2912 generally can be used.

If the second descriptor 2912 can be used to distinguish the firstperson 3102 from the candidates, the tracking subsystem 2900 proceeds tostep 3222, and the tracking subsystem 2900 determines the identifier2908 for the first person 3102 based on the second descriptor 2912 andupdates the candidate list 2906 for the first person 3102 accordingly.The identifier 2908 for the first person 3102 may be determined asdescribed above with respect to step 3214 (e.g., by calculating a cosinesimilarity value between a vector corresponding to the first descriptor2912 and previously determined vectors associated with the predetermineddescriptors 2910). The tracking subsystem 2900 then proceeds to step3216 described above to update identifiers 2908 (i.e., via candidatelists 2906) of other tracked people 3104, 3106 as appropriate.

Otherwise, if the second descriptor 2912 cannot be used to distinguishthe first person 3102 from the candidates, the tracking subsystem 2900proceeds to step 3224, and the tracking subsystem 2900 determines adescriptor 2912 for all of the first person 3102 and all of thecandidates. In other words, a measured descriptor 2912 is determined forall people associated with the identifiers 2908 appearing in thecandidate list 2906 of the first person 3102 (e.g., as described abovewith respect to FIG. 31C). At step 3226, the tracking subsystem 2900compares the second descriptor 2912 to predetermined descriptors 2910associated with all people related to the candidate list 2906 of thefirst person 3102. For instance, the tracking subsystem 2900 maydetermine a second cosine similarity value between a second data vectordetermined using an artificial neural network and each correspondingvector from the predetermined descriptor values 2910 for the candidates(e.g., as illustrated in FIG. 31C, described above). The trackingsubsystem 2900 then proceeds to step 3228 to determine and update theidentifiers 2908 of all candidates based on the comparison at step 3226before continuing to track people 3102, 3104, 3106 in the space 102(e.g., by returning to step 3204).

Modifications, additions, or omissions may be made to method 3200depicted in FIG. 32 . Method 3200 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 2900 (e.g.,by server 106 and/or client(s) 105) or components thereof performingsteps, any suitable system or components of the system may perform oneor more steps of the method 3200.

Action Detection for Assigning Items to the Correct Person

As described above with respect to FIGS. 12-15 when a weight event isdetected at a rack 112, the item associated with the activated weightsensor 110 may be assigned to the person nearest the rack 112. However,in some cases, two or more people may be near the rack 112 and it maynot be clear who picked up the item. Accordingly, further action may berequired to properly assign the item to the correct person.

In some embodiments, a cascade of algorithms (e.g., from more simpleapproaches based on relatively straightforwardly determined imagefeatures to more complex strategies involving artificial neuralnetworks) may be employed to assign an item to the correct person. Thecascade may be triggered, for example, by (i) the proximity of two ormore people to the rack 112, (ii) a hand crossing into the zone (or a“virtual curtain”) adjacent to the rack (e.g., see zone 3324 of FIG. 33Band corresponding description below) and/or, (iii) a weight signalindicating an item was removed from the rack 112. When it is initiallyuncertain who picked up an item, a unique contour-based approach may beused to assign an item to the correct person. For instance, if twopeople may be reaching into a rack 112 to pick up an item, a contour maybe “dilated” from a head height to a lower height in order to determinewhich person's arm reached into the rack 112 to pick up the item.However, if the results of this efficient contour-based approach do notsatisfy certain confidence criteria, a more computationally expensiveapproach (e.g., involving neural network-based pose estimation) may beused. In some embodiments, the tracking system 100, upon detecting thatmore than one person may have picked up an item, may store a set ofbuffer frames that are most likely to contain useful information foreffectively assigning the item to the correct person. For instance, thestored buffer frames may correspond to brief time intervals when aportion of a person enters the zone adjacent to a rack 112 (e.g., zone3324 of FIG. 33B, described above) and/or when the person exits thiszone.

However, in some cases, it may still be difficult or impossible toassign an item to a person even using more advance artificial neuralnetwork-based pose estimation techniques. In these cases, the trackingsystem 100 may store further buffer frames in order to track the itemthrough the space 102 after it exits the rack 112. When the item comesto a stopped position (e.g., with a sufficiently low velocity), thetracking system 100 determines which person is closer to the stoppeditem, and the item is generally assigned to the nearest person. Thisprocess may be repeated until the item is confidently assigned to thecorrect person.

FIG. 33A illustrates an example scenario in which a first person 3302and a second person 3304 are near a rack 112 storing items 3306 a-c.Each item 3306 a-c is stored on corresponding weight sensors 110 a-c. Asensor 108, which is communicatively coupled to the tracking subsystem3300 (i.e., to the server 106 and/or client(s) 105), generates atop-view depth image 3308 for a field-of-view 3310 which includes therack 112 and people 3302, 3304. The top-view depth image 3308 includes arepresentation 112 a of the rack 112 and representations 3302 a, 3304 aof the first and second people 3302, 3304, respectively. The rack 112(e.g., or its representation 112 a) may be divided into three zones 3312a-c which correspond to the locations of weight sensors 110 a-c and theassociated items 3306 a-c, respectively.

In this example scenario, one of the people 3302, 3304 picks up an item3306 c from weight sensor 110 c, and tracking subsystem 3300 receives atrigger signal 3314 indicating an item 3306 c has been removed from therack 112. The tracking subsystem 3300 includes the client(s) 105 andserver 106 described above with respect to FIG. 1 . The trigger signal3314 may indicate the change in weight caused by the item 3306 c beingremoved from sensor 110 c. After receiving the signal 3314, the server106 accesses the top-view image 3308, which may correspond to a time at,just prior to, and/or just following the time the trigger signal 3314was received. In some embodiments, the trigger signal 3314 may also oralternatively be associated with the tracking system 100 detecting aperson 3302, 3304 entering a zone adjacent to the rack (e.g., asdescribed with respect to the “virtual curtain” of FIGS. 12-15 aboveand/or zone 3324 described in greater detail below) to determine towhich person 3302, 3304 the item 3306 c should be assigned. Sincerepresentations 3302 a and 3304 a indicate that both people 3302, 3304are near the rack 112, further analysis is required to assign item 3306c to the correct person 3302, 3304. Initially, the tracking system 100may determine if an arm of either person 3302 or 3304 may be reachingtoward zone 3312 c to pick up item 3306 c. However, as shown in regions3316 and 3318 in image 3308, a portion of both representations 3302 a,3304 a appears to possibly be reaching toward the item 3306 c in zone3312 c. Thus, further analysis is required to determine whether thefirst person 3302 or the second person 3304 picked up item 3306 c.

Following the initial inability to confidently assign item 3306 c to thecorrect person 3302, 3304, the tracking system 100 may use acontour-dilation approach to determine whether person 3302 or 3304picked up item 3306 c. FIG. 33B illustrates an implementation of acontour-dilation approach to assigning item 3306 c to the correct person3302 or 3304. In general, contour dilation involves iterative dilationof a first contour associated with the first person 3302 and a secondcontour associated with the second person 3304 from a first smallerdepth to a second larger depth. The dilated contour that crosses intothe zone 3324 adjacent to the rack 112 first may correspond to theperson 3302, 3304 that picked up the item 3306 c. Dilated contours mayneed to satisfy certain criteria to ensure that the results of thecontour-dilation approach should be used for item assignment. Forexample, the criteria may include a requirement that a portion of acontour entering the zone 3324 adjacent to the rack 112 is associatedwith either the first person 3302 or the second person 3304 within amaximum number of iterative dilations, as is described in greater detailwith respect to the contour-detection views 3320, 3326, 3328, and 3332shown in FIG. 33B. If these criteria are not satisfied, another methodshould be used to determine which person 3302 or 3304 picked up item3306 c.

FIG. 33B shows a view 3320, which includes a contour 3302 b detected ata first depth in the top-view image 3308. The first depth may correspondto an approximate head height of a typical person 3322 expected to betracked in the space 102, as illustrated in FIG. 33B. Contour 3302 bdoes not enter or contact the zone 3324 which corresponds to thelocation of a space adjacent to the front of the rack 112 (e.g., asdescribed with respect to the “virtual curtain” of FIGS. 12-15 above).Therefore, the tracking system 100 proceeds to a second depth in image3308 and detects contours 3302 c and 3304 b shown in view 3326. Thesecond depth is greater than the first depth of view 3320. Since neitherof the contours 3302 c or 3304 b enter zone 3324, the tracking system100 proceeds to a third depth in the image 3308 and detects contours3302 d and 3304 c, as shown in view 3328. The third depth is greaterthan the second depth, as illustrated with respect to person 3322 inFIG. 33B.

In view 3328, contour 3302 d appears to enter or touch the edge of zone3324. Accordingly, the tracking system 100 may determine that the firstperson 3302, who is associated with contour 3302 d, should be assignedthe item 3306 c. In some embodiments, after initially assigning the item3306 c to person 3302, the tracking system 100 may project an “armsegment” 3330 to determine whether the arm segment 3330 enters theappropriate zone 3312 c that is associated with item 3306 c. The armsegment 3330 generally corresponds to the expected position of theperson's extended arm in the space occluded from view by the rack 112.If the location of the projected arm segment 3330 does not correspondwith an expected location of item 3306 c (e.g., a location within zone3312 c), the item is not assigned to (or is unassigned from) the firstperson 3302.

Another view 3332 at a further increased fourth depth shows a contour3302 e and contour 3304 d. Each of these contours 3302 e and 3304 dappear to enter or touch the edge of zone 3324. However, since thedilated contours associated with the first person 3302 (reflected incontours 3302 b-e) entered or touched zone 3324 within fewer iterations(or at a smaller depth) than did the dilated contours associated withthe second person 3304 (reflected in contours 3304 b-d), the item 3306 cis generally assigned to the first person 3302. In general, in order forthe item 3306 c to be assigned to one of the people 3302, 3304 usingcontour dilation, a contour may need to enter zone 3324 within a maximumnumber of dilations (e.g., or before a maximum depth is reached). Forexample, if the item 3306 c was not assigned by the fourth depth, thetracking system 100 may have ended the contour-dilation method and movedon to another approach to assigning the item 3306 c, as described below.

In some embodiments the contour-dilation approach illustrated in FIG.33B fails to correctly assign item 3306 c to the correct person 3302,3304. For example, the criteria described above may not be satisfied(e.g., a maximum depth or number of iterations may be exceeded) ordilated contours associated with the different people 3302 or 3304 maymerge, rendering the results of contour-dilation unusable. In suchcases, the tracking system 100 may employ another strategy to determinewhich person 3302, 3304 c picked up item 3306 c. For example, thetracking system 100 may use a pose estimation algorithm to determine apose of each person 3302, 3304.

FIG. 33C illustrates an example output of a pose-estimation algorithmwhich includes a first “skeleton” 3302 f for the first person 3302 and asecond “skeleton” 3304 e for the second person 3304. In this example,the first skeleton 3302 f may be assigned a “reaching pose” because anarm of the skeleton appears to be reaching outward. This reaching posemay indicate that the person 3302 is reaching to pick up item 3306 c. Incontrast, the second skeleton 3304 e does not appear to be reaching topick up item 3306 c. Since only the first skeleton 3302 f appears to bereaching for the item 3306 c, the tracking system 100 may assign theitem 3306 c to the first person 3302. If the results of pose estimationwere uncertain (e.g., if both or neither of the skeletons 3302 f, 3304 eappeared to be reaching for item 3306 c), a different method of itemassignment may be implemented by the tracking system 100 (e.g., bytracking the item 3306 c through the space 102, as described below withrespect to FIGS. 36-37 ).

FIG. 34 illustrates a method 3400 for assigning an item 3306 c to aperson 3302 or 3304 using the tracking system 100. The method 3400 maybegin at step 3402 where the tracking system 100 receives an image feedcomprising frames of top-view images generated by the sensor 108 andweight measurements from weight sensors 110 a-c.

At step 3404, the tracking system 100 detects an event associated withpicking up an item 33106 c. In general, the event may be based on aportion of a person 3302, 3304 entering the zone adjacent to the rack112 (e.g., zone 3324 of FIG. 33B) and/or a change of weight associatedwith the item 33106 c being removed from the corresponding weight sensor110 c.

At step 3406, in response to detecting the event at step 3404, thetracking system 100 determines whether more than one person 3302, 3304may be associated with the detected event (e.g., as in the examplescenario illustrated in FIG. 33A, described above). For example, thisdetermination may be based on distances between the people and the rack112, an inter-person distance between the people, a relative orientationbetween the people and the rack 112 (e.g., a person 3302, 3304 notfacing the rack 112 may not be a candidate for picking up the item 33106c). If only one person 3302, 3304 may be associated with the event, thatperson 3302, 3304 is associated with the item 3306 c at step 3408. Forexample, the item 3306 c may be assigned to the nearest person 3302,3304, as described with respect to FIGS. 12-14 above.

At step 3410, the item 3306 c is assigned to the person 3302, 3304determined to be associated with the event detected at step 3404. Forexample, the item 3306 c may be added to a digital cart associated withthe person 3302, 3304. Generally, if the action (i.e., picking up theitem 3306 c) was determined to have been performed by the first person3302, the action (and the associated item 3306 c) is assigned to thefirst person 3302, and, if the action was determined to have beenperformed by the second person 3304, the action (and associated item3306 c) is assigned to the second person 3304.

Otherwise, if, at step 3406, more than one person 3302, 3304 may beassociated with the detected event, a select set of buffer frames oftop-view images generated by sensor 108 may be stored at step 3412. Insome embodiments, the stored buffer frames may include only three orfewer frames of top-view images following a triggering event. Thetriggering event may be associated with the person 3302, 3304 enteringthe zone adjacent to the rack 112 (e.g., zone 3324 of FIG. 33B), theportion of the person 3302, 3304 exiting the zone adjacent to the rack112 (e.g., zone 3324 of FIG. 33B), and/or a change in weight determinedby a weight sensor 110 a-c. In some embodiments, the buffer frames mayinclude image frames from the time a change in weight was reported by aweight sensor 110 until the person 3302, 3304 exits the zone adjacent tothe rack 112 (e.g., zone 3324 of FIG. 33B). The buffer frames generallyinclude a subset of all possible frames available from the sensor 108.As such, by storing, and subsequently analyzing, only these storedbuffer frames (or a portion of the stored buffer frames), the trackingsystem 100 may assign actions (e.g., and an associated item 106 a-c) toa correct person 3302, 3304 more efficiently (e.g., in terms of the useof memory and processing resources) than was possible using previoustechnology.

At step 3414, a region-of-interest from the images may be accessed. Forexample, following storing the buffer frames, the tracking system 100may determine a region-of-interest of the top-view images to retain. Forexample, the tracking system 100 may only store a region near the centerof each view (e.g., region 3006 illustrated in FIG. 30 and describedabove).

At step 3416, the tracking system 100 determines, using at least one ofthe buffer frames stored at step 3412 and a first action-detectionalgorithm, whether an action associated with the detected event wasperformed by the first person 3302 or the second person 3304. The firstaction-detection algorithm is generally configured to detect the actionbased on characteristics of one or more contours in the stored bufferframes. As an example, the first action-detection algorithm may be thecontour-dilation algorithm described above with respect to FIG. 33B. Anexample implementation of a contour-based action-detection method isalso described in greater detail below with respect to method 3500illustrated in FIG. 35 . In some embodiments, the tracking system 100may determine a subset of the buffer frames to use with the firstaction-detection algorithm. For example, the subset may correspond towhen the person 3302, 3304 enters the zone adjacent to the rack 112(e.g., zone 3324 illustrated in FIG. 33B).

At step 3418, the tracking system 100 determines whether results of thefirst action-detection algorithm satisfy criteria indicating that thefirst algorithm is appropriate for determining which person 3302, 3304is associated with the event (i.e., picking up item 3306 c, in thisexample). For example, for the contour-dilation approach described abovewith respect to FIG. 33B and below with respect to FIG. 35 , thecriteria may be a requirement to identify the person 3302, 3304associated with the event within a threshold number of dilations (e.g.,before reaching a maximum depth). Whether the criteria are satisfied atstep 3416 may be based at least in part on the number of iterationsrequired to implement the first action-detection algorithm. If thecriteria are satisfied at step 3418, the tracking system 100 proceeds tostep 3410 and assigns the item 3306 c to the person 3302, 3304associated with the event determined at step 3416.

However, if the criteria are not satisfied at step 3418, the trackingsystem 100 proceeds to step 3420 and uses a different action-detectionalgorithm to determine whether the action associated with the eventdetected at step 3404 was performed by the first person 3302 or thesecond person 3304. This may be performed by applying a secondaction-detection algorithm to at least one of the buffer frames selectedat step 3412. The second action-detection algorithm may be configured todetect the action using an artificial neural network. For example, thesecond algorithm may be a pose estimation algorithm used to determinewhether a pose of the first person 3302 or second person 3304corresponds to the action (e.g., as described above with respect to FIG.33C). In some embodiments, the tracking system 100 may determine asecond subset of the buffer frames to use with the second actiondetection algorithm. For example, the subset may correspond to the timewhen the weight change is reported by the weight sensor 110. The pose ofeach person 3302, 3304 at the time of the weight change may provide agood indication of which person 3302, 3304 picked up the item 3306 c.

At step 3422, the tracking system 100 may determine whether the secondalgorithm satisfies criteria indicating that the second algorithm isappropriate for determining which person 3302, 3304 is associated withthe event (i.e., with picking up item 3306 c). For example, if the poses(e.g., determined from skeletons 3302 f and 3304 e of FIG. 33C,described above) of each person 3302, 3304 still suggest that eitherperson 3302, 3304 could have picked up the item 3306 c, the criteria maynot be satisfied, and the tracking system 100 proceeds to step 3424 toassign the object using another approach (e.g., by tracking the movementof the item 3306 a-c through the space 102, as described in greaterdetail below with respect to FIGS. 36 and 37 ).

Modifications, additions, or omissions may be made to method 3400depicted in FIG. 34 . Method 3400 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 100 orcomponents thereof performing steps, any suitable system or componentsof the system may perform one or more steps of the method 3400.

As described above, the first action-detection algorithm of step 3416may involve iterative contour dilation to determine which person 3302,3304 is reaching to pick up an item 3306 a-c from rack 112. FIG. 35illustrates an example method 3500 of contour dilation-based itemassignment. The method 3500 may begin from step 3416 of FIG. 34 ,described above, and proceed to step 3502. At step 3502, the trackingsystem 100 determines whether a contour is detected at a first depth(e.g., the first depth of FIG. 33B described above). For example, in theexample illustrated in FIG. 33B, contour 3302 b is detected at the firstdepth. If a contour is not detected, the tracking system 100 proceeds tostep 3504 to determine if the maximum depth (e.g., the fourth depth ofFIG. 33B) has been reached. If the maximum depth has not been reached,the tracking system 100 iterates (i.e., moves) to the next depth in theimage at step 3506. Otherwise, if the maximum depth has been reached,method 3500 ends.

If at step 3502, a contour is detected, the tracking system proceeds tostep 3508 and determines whether a portion of the detected contouroverlaps, enters, or otherwise contacts the zone adjacent to the rack112 (e.g., zone 3324 illustrated in FIG. 33B). In some embodiments, thetracking system 100 determines if a projected arm segment (e.g., armsegment 3330 of FIG. 33B) of a contour extends into an appropriate zone3312 a-c of the rack 112. If no portion of the contour extends into thezone adjacent to the rack 112, the tracking system 100 determineswhether the maximum depth has been reached at step 3504. If the maximumdepth has not been reached, the tracking system 100 iterates to the nextlarger depth and returns to step 3502.

At step 3510, the tracking system 100 determines the number ofiterations (i.e., the number of times step 3506 was performed) beforethe contour was determined to have entered the zone adjacent to the rack112 at step 3508. At step 3512, this number of iterations is compared tothe number of iterations for a second (i.e., different) detectedcontour. For example, steps 3502 to 35010 may be repeated to determinethe number of iterations (at step 3506) for the second contour to enterthe zone adjacent to the rack 112. If the number of iterations is lessthan that of the second contour, the item is assigned to the firstperson 3302 at step 3514. Otherwise, the item may be assigned to thesecond person 3304 at step 3516. For example, as described above withrespect to FIG. 33B, the first dilated contours 3302 b-e entered thezone 3324 adjacent to the rack 112 within fewer iterations than did thesecond dilated contours 3304 b. In this example, the item is assigned tothe person 3302 associated with the first contour 3302 b-d.

In some embodiments, a dilated contour (i.e., the contour generated viatwo or more passes through step 3506) must satisfy certain criteria inorder for it to be used for assigning an item. For instance, a contourmay need to enter the zone adjacent to the rack within a maximum numberof dilations (e.g., or before a maximum depth is reached), as describedabove. As another example, a dilated contour may need to include lessthan a threshold number of pixels. If a contour is too large it may be a“merged contour” that is associated with two closely spaced people (seeFIG. 22 and the corresponding description above).

Modifications, additions, or omissions may be made to method 3500depicted in FIG. 35 . Method 3500 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 100 orcomponents thereof performing steps, any suitable system or componentsof the system may perform one or more steps of the method 3500.

Item Tracking-Based Item Assignment

As described above, in some cases, an item 3306 a-c cannot be assignedto the correct person even using a higher-level algorithm such as theartificial neural network-based pose estimation described above withrespect to FIGS. 33C and 34 . In these cases, the position of the item3306 c after it exits the rack 112 may be tracked in order to assign theitem 3306 c to the correct person 3302, 3304. In some embodiments, thetracking system 100 does this by tracking the item 3306 c after it exitsthe rack 112, identifying a position where the item stops moving, anddetermining which person 3302, 3304 is nearest to the stopped item 3306c. The nearest person 3302, 3304 is generally assigned the item 3306 c.

FIGS. 36A,B illustrate this item tracking-based approach to itemassignment. FIG. 36A shows a top-view image 3602 generated by a sensor108. FIG. 36B shows a plot 3620 of the item's velocity 3622 over time.As shown in FIG. 36A, image 3602 includes a representation of a person3604 holding an item 3606 which has just exited a zone 3608 adjacent toa rack 112. Since a representation of a second person 3610 may also havebeen associated with picking up the item 3606, item-based tracking isrequired to properly assign the item 3606 to the correct person 3604,3610 (e.g., as described above with respect people 3302, 3304 and item3306 c for FIGS. 33-35 ). Tracking system 100 may (i) track the positionof the item 3606 over time after the item 3606 exits the rack 112, asillustrated in tracking views 3610 and 3616, and (ii) determine thevelocity of the item 3606, as shown in curve 3622 of plot 3620 in FIG.36B. The velocity 3622 shown in FIG. 36B is zero at the inflectionpoints corresponding to a first stopped time (t_(stopped,1)) and asecond stopped time (t_(stopped,2)). More generally, the time when theitem 3606 is stopped may correspond to a time when the velocity 3622 isless than a threshold velocity 3624.

Tracking view 3612 of FIG. 36A shows the position 3604 a of the firstperson 3604, a position 3606 a of item 3606, and a position 3610 a ofthe second person 3610 at the first stopped time. At the first stoppedtime (t_(stopped,1)) the positions 3604 a, 3610 a are both near theposition 3606 a of the item 3606. Accordingly, the tracking system 100may not be able to confidently assign item 3606 to the correct person3604 or 3610. Thus, the tracking system 100 continues to track the item3606. Tracking view 3614 shows the position 3604 a of the first person3604, the position 3606 a of the item 3606, and the position 3610 a ofthe second person 3610 at the second stopped time (t_(stopped,2)). Sinceonly the position 3604 a of the first person 3604 is near the position3606 a of the item 3606, the item 3606 is assigned to the first person3604.

More specifically, the tracking system 100 may determine, at eachstopped time, a first distance 3626 between the stopped item 3606 andthe first person 3604 and a second distance 3628 between the stoppeditem 3606 and the second person 3610. Using these distances 3626, 3628,the tracking system 100 determines whether the stopped position of theitem 3606 in the first frame is nearer the first person 3604 or nearerthe second person 3610 and whether the distance 3626, 3628 is less thana threshold distance 3630. At the first stopped time of view 3612, bothdistances 3626, 3628 are less than the threshold distance 3630. Thus,the tracking system 100 cannot reliably determine which person 3604,3610 should be assigned the item 3606. In contrast, at the secondstopped time of view 3614, only the first distance 3626 is less than thethreshold distance 3630. Therefore, the tracking system may assign theitem 3606 to the first person 3604 at the second stopped time.

FIG. 37 illustrates an example method 3700 of assigning an item 3606 toa person 3604 or 3610 based on item tracking using tracking system 100.Method 3700 may begin at step 3424 of method 3400 illustrated in FIG. 34and described above and proceed to step 3702. At step 3702, the trackingsystem 100 may determine that item tracking is needed (e.g., because theaction-detection based approaches described above with respect to FIGS.33-35 were unsuccessful). At step 3504, the tracking system 100 storesand/or accesses buffer frames of top-view images generated by sensor108. The buffer frames generally include frames from a time periodfollowing a portion of the person 3604 or 3610 exiting the zone 3608adjacent to the rack 11236.

At step 3706, the tracking system 100 tracks, in the stored frames, aposition of the item 3606. The position may be a local pixel positionassociated with the sensor 108 (e.g., determined by client 105) or aglobal physical position in the space 102 (e.g., determined by server106 using an appropriate homography). In some embodiments, the item 3606may include a visually observable tag that can be viewed by the sensor108 and detected and tracked by the tracking system 100 using the tag.In some embodiments, the item 3606 may be detected by the trackingsystem 100 using a machine learning algorithm. To facilitate detectionof many item types under a broad range of conditions (e.g., differentorientations relative to the sensor 108, different lighting conditions,etc.), the machine learning algorithm may be trained using syntheticdata (e.g., artificial image data that can be used to train thealgorithm).

At step 3708, the tracking system 100 determines whether a velocity 3622of the item 3606 is less than a threshold velocity 3624. For example,the velocity 3622 may be calculated, based on the tracked position ofthe item 3606. For instance, the distance moved between frames may beused to calculate a velocity 3622 of the item 3606. A particle filtertracker (e.g., as described above with respect to FIGS. 24-26 ) may beused to calculate item velocity 3622 based on estimated future positionsof the item. If the item velocity 3622 is below the threshold 3624, thetracking system 100 identifies, a frame in which the velocity 3622 ofthe item 3606 is less than the threshold velocity 3624 and proceeds tostep 3710. Otherwise, the tracking system 100 continues to track theitem 3606 at step 3706.

At step 3710, the tracking system 100 determines, in the identifiedframe, a first distance 3626 between the stopped item 3606 and a firstperson 3604 and a second distance 3628 between the stopped item 3606 anda second person 3610. Using these distances 3626, 3628, the trackingsystem 100 determines, at step 3712, whether the stopped position of theitem 3606 in the first frame is nearer the first person 3604 or nearerthe second person 3610 and whether the distance 3626, 3628 is less thana threshold distance 3630. In general, in order for the item 3606 to beassigned to the first person 3604, the item 3606 should be within thethreshold distance 3630 from the first person 3604, indicating theperson is likely holding the item 3606, and closer to the first person3604 than to the second person 3610. For example, at step 3712, thetracking system 100 may determine that the stopped position is a firstdistance 3626 away from the first person 3604 and a second distance 3628away from the second person 3610. The tracking system 100 may determinean absolute value of a difference between the first distance 3626 andthe second distance 3628 and may compare the absolute value to athreshold distance 3630. If the absolute value is less than thethreshold distance 3630, the tracking system returns to step 3706 andcontinues tracking the item 3606. Otherwise, the tracking system 100 isgreater than the threshold distance 3630 and the item 3606 issufficiently close to the first person 3604, the tracking systemproceeds to step 3714 and assigns the item 3606 to the first person3604.

Modifications, additions, or omissions may be made to method 3700depicted in FIG. 37 . Method 3700 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While at times discussed as tracking system 100 orcomponents thereof performing steps, any suitable system or componentsof the system may perform one or more steps of the method 3700.Hardware Configuration

FIG. 38 is an embodiment of a device 3800 (e.g. a server 106 or a client105) configured to track objects and people within a space 102. Thedevice 3800 comprises a processor 3802, a memory 3804, and a networkinterface 3806. The device 3800 may be configured as shown or in anyother suitable configuration.

The processor 3802 comprises one or more processors operably coupled tothe memory 3804. The processor 3802 is any electronic circuitryincluding, but not limited to, state machines, one or more centralprocessing unit (CPU) chips, logic units, cores (e.g. a multi-coreprocessor), field-programmable gate array (FPGAs), application specificintegrated circuits (ASICs), or digital signal processors (DSPs). Theprocessor 3802 may be a programmable logic device, a microcontroller, amicroprocessor, or any suitable combination of the preceding. Theprocessor 3802 is communicatively coupled to and in signal communicationwith the memory 3804. The one or more processors are configured toprocess data and may be implemented in hardware or software. Forexample, the processor 3802 may be 8-bit, 16-bit, 32-bit, 64-bit or ofany other suitable architecture. The processor 3802 may include anarithmetic logic unit (ALU) for performing arithmetic and logicoperations, processor registers that supply operands to the ALU andstore the results of ALU operations, and a control unit that fetchesinstructions from memory and executes them by directing the coordinatedoperations of the ALU, registers and other components.

The one or more processors are configured to implement variousinstructions. For example, the one or more processors are configured toexecute instructions to implement a tracking engine 3808. In this way,processor 3802 may be a special purpose computer designed to implementthe functions disclosed herein. In an embodiment, the tracking engine3808 is implemented using logic units, FPGAs, ASICs, DSPs, or any othersuitable hardware. The tracking engine 3808 is configured operate asdescribed in FIGS. 1-18 . For example, the tracking engine 3808 may beconfigured to perform the steps of methods 200, 600, 800, 1000, 1200,1500, 1600, 1700, 5900, 6000, 6500, 6800, 7000, 7200, and 7400, asdescribed in FIGS. 2, 6, 8, 10, 12, 15, 16, 17, 59, 60, 65, 68, 70, 72,and 74 , respectively.

The memory 3804 comprises one or more disks, tape drives, or solid-statedrives, and may be used as an over-flow data storage device, to storeprograms when such programs are selected for execution, and to storeinstructions and data that are read during program execution. The memory3804 may be volatile or non-volatile and may comprise read-only memory(ROM), random-access memory (RAM), ternary content-addressable memory(TCAM), dynamic random-access memory (DRAM), and static random-accessmemory (SRAM).

The memory 3804 is operable to store tracking instructions 3810,homographies 118, marker grid information 716, marker dictionaries 718,pixel location information 908, adjacency lists 1114, tracking lists1112, digital carts 1410, item maps 1308, disparity mappings 7308,and/or any other data or instructions. The tracking instructions 3810may comprise any suitable set of instructions, logic, rules, or codeoperable to execute the tracking engine 3808.

The homographies 118 are configured as described in FIGS. 2-5B. Themarker grid information 716 is configured as described in FIGS. 6-7 .The marker dictionaries 718 are configured as described in FIGS. 6-7 .The pixel location information 908 is configured as described in FIGS.8-9 . The adjacency lists 1114 are configured as described in FIGS.10-11 . The tracking lists 1112 are configured as described in FIGS.10-11 . The digital carts 1410 are configured as described in FIGS.12-18 . The item maps 1308 are configured as described in FIGS. 12-18 .The disparity mappings 7308 are configured as described in FIGS. 72 and73 .

The network interface 3806 is configured to enable wired and/or wirelesscommunications. The network interface 3806 is configured to communicatedata between the device 3800 and other, systems, or domain. For example,the network interface 3806 may comprise a WIFI interface, a LANinterface, a WAN interface, a modem, a switch, or a router. Theprocessor 3802 is configured to send and receive data using the networkinterface 3806. The network interface 3806 may be configured to use anysuitable type of communication protocol as would be appreciated by oneof ordinary skill in the art.

Item Assignment Based on Angled-View Images

As described above, an item may be assigned to an appropriate personbased on proximity to a rack 112 and activation of a weight sensor 110on which the item is known to be placed (see, e.g., FIGS. 12-15 and thecorresponding description above). In cases where two or more people maybe near the rack 112 and it may not be clear who picked up the item,further action may be taken to properly assign the item to the correctperson, as described with respect to FIGS. 33A-37 . For example, in someembodiments, a cascade of algorithms may be employed to assign an itemto the correct person. In some cases, the assignment of an item to theappropriate person may still be difficult using the approaches describedabove. In such cases, the tracking system 100 may store further bufferframes in order to track the item through the space 102 after it exitsthe rack 112, as described above with respect to FIGS. 36A-B and 37.When the item comes to a stopped position (e.g., with a sufficiently lowvelocity), the tracking system 100 may determine which person is closerto the stopped item, and the item may be assigned to the nearest person.This process may be repeated until the item is assigned with confidenceto the correct person.

To handle instances where a weight sensor 110 is not present, reducereliance on item tracking-based approaches to item assignment, anddecrease the consumption of processing resources associated withcontinued item tracking and identification, a new approach, which isdescribed further below with respect to FIGS. 39-44 , may be used inwhich angled-view images of a portion of a rack 112 are captured andused to efficiently and reliably assign items to the appropriate person.

FIG. 39 illustrates an example scenario 3900 in which angled-view images3918 captured by an angled-view sensor 3914 are used to assign aselected item 3924 a-i to a person 3902 moving about the space 102 ofFIG. 1 . The angled-view images 3918 are provided to a trackingsubsystem 3910 (e.g., the server 106 and/or client(s) 105 describedabove) and used to detect an interaction between the person 3902 and therack 112, identify an item 3924 a-i interacted with by the person 3902,and determine whether the interaction corresponds to the person 3902taking the item 3924 a-i from the rack 112 or placing the item 3924 a-ion the rack 112. This information is used to appropriately assign items3924 a-i to the person 3902, for example, by updating the digitalshopping cart 3926 associated an identifier 3928 of the person 3902(e.g., a user ID, account number, name, or the like) to an identifier3930 of the selected item 3924 a-i (e.g., a product number, productname, or the like). The digital shopping cart 3926 may be the same orsimilar to the digital shopping carts (e.g., digital cart 1410)described above with respect to FIGS. 12-18 .

In one embodiment, the identifier 3928 of the person 3902 may be a localidentifier that is assigned by a sensor 108. In this case, the trackingsystem 100 may use a global identifier that is associated with theperson 3902. The tracking system 100 is configured to store associationsbetween local identifiers from different sensors 108 and globalidentifiers for a person. For example, the tracking system 100 may beconfigured to receive a local identifier for a person and identifiersfor items the person is removing from a rack 112. The tracking system100 uses a mapping (e.g. a look-up table) to identify a globalidentifier for a person based on their local identifier from a sensor108. After identifying the global identifier for the person, thetracking system 100 may update the digital cart 1410 that is associatedwith the person by adding or removing items from their digital cart1410.

In the example scenario of FIG. 39 , the person 3902 is approaching therack 112 at an initial time, t₁, and is near the rack 112 at asubsequent time, t₂. The rack 112 stores items 3924 a-i. A top-viewsensor 3904, which is communicatively coupled to the tracking subsystem3910 (i.e., to the server 106 and/or client(s) 105 described in greaterdetail with respect to FIG. 1 above), generates top-view images 3908 fora field-of-view 3906. The top-view images 3908 may be any type of image(e.g., a color image, depth image, and/or the like). The field-of-view3906 of the top-view sensor 3904 may include the rack 112 and/or aregion adjacent to the rack 112. The top-view sensor 3904 may be asensor 108 described above with respect to FIG. 1 . A top-view image3908 captured at time t₁ includes a representation of the person 3902and, optionally, the rack 112 (e.g., depending on the extent of thefield-of-view 3906). The tracking subsystem 3910 is configured todetermine, based on the top-view image 3906, whether the person 3902 iswithin a threshold distance 3912 of the rack 112 (e.g., by determining aphysical position of the person 3902 using a homography 118, asdescribed with respect to FIGS. 2-7 and determining if this physicalposition is within the threshold distance 3912 of a predefined positionof the rack 112).

If the person 3902 is determined to be within the threshold distance3912 of the rack 112, the tracking subsystem 3910 may begin receivingangled-view images 3918 captured by the angled-view sensor 3914. Forexample, after the person 3902 is determined to be within the thresholddistance 3912 of the rack 112, the tracking subsystem 3910 may instructthe angled-view sensor 3914 to begin capturing angled-view images 3918.As such, the top-view images 3908 may be used to determine when aproximity trigger (e.g., the proximity trigger 4002 of FIG. 40 ) causesthe start of angled-view image 3918 acquisition and/or processing. Theangled-view sensor 3914 generates angled-view images 3918 for afield-of-view 3916. The field-of-view 3916 generally includes at least aportion of the rack 112 (e.g., the field-of-view 3916 may include a viewinto the shelves 3920 a-c of the rack 112 on which items 3924 a-i areplaced). The angled-view sensor 3914 may include one or more sensors,such as a color camera, a depth camera, an infrared sensor, and/or thelike. The angled-view sensor 3914 may be a sensor 108 described withrespect to FIG. 1 , or a sensor 108 b described with respect to FIG. 22.

An angled-view image 3918 captured at time t₂ includes a representationof the person 3902 and the portion of the rack 112 included in thefield-of-view 3916 of the angled-view sensor 3914. As described ingreater detail below with respect to FIGS. 40-44 , the trackingsubsystem 3910 is configured to determine whether the person 3902interacts with the rack 112 and/or one or more items 3924 a-i stored onthe rack 112 (see, e.g., the item localization and event triggerinstructions 4004 of FIG. 40 and the corresponding description below),identify item(s) 3924 a-i interacted with by the person 3902 (see, e.g.,the item identification instructions 4012 of FIG. 40 and thecorresponding description below), and determine whether the identifieditem(s) 3924 a-i was/were removed from or placed on the rack 112 (see,e.g., the activity recognition instructions 4016 of FIG. 40 and thecorresponding description below). This information is used toappropriately assign selected items 3924 a-i to the person 3902, forexample, by updating the digital shopping cart 3926 associated with theperson 3902.

In some embodiments, one or more of the shelves 3920 a-c of the rack 112includes a visible marker 3922 a-c (e.g., a series of visible shapes orany other marker at a predefined location on the shelf 3920 a-c). Thetracking subsystem 3910 may detect the markers 3922 a-c in angled-viewimage 3918 of the shelves 3920 a-c and determine the pixel positions ofthe shelves 3920 a-c in the images 3918 based on these markers 3922 a-c.In some cases, pixel positions of the shelves 3920 a-c in the images3918 are predefined (e.g., without using markers 3922 a-c). For example,the tracking subsystem 3910 may determine a predefined shelf position inthe images 3918 for one or more of the shelves 3920 a-c. Thisinformation may facilitate improved detection of an interaction betweena person 3902 and a given shelf 3920 a-c, as described further belowwith respect to FIGS. 40-44 .

In some embodiments, the rack 112 includes one or more weight sensors110 a-i. However, efficient and reliable assignment of item(s) 3924 a-ito a person 3902 can be achieved without weight sensors 110 a-i. Assuch, in some embodiments, the rack 112 does not include weight sensors110 a-i. In embodiments in which the rack 112 includes one or moreweight sensors 110 a-i, each weight sensor 110 a-i may store items 3924a-i of the same type. For instance, a first weight sensor 110 a may beassociated with items 3924 a of a first type (e.g., a particular brandand size of product), a second weight sensor 110 b may be associatedwith items 3924 b of a second type, and so on (see FIG. 13 ). Changes inweight measured by weight sensors 110 a-i may provide further insightinto when the person 3902 interacts with the rack 112 (e.g., based on atime when a change of weight is detected by a weight sensor 110 a-i) andwhich item(s) 3924 a-i the person 3902 interacts with on the rack 112(e.g., based on knowledge of which item 3924 a-i should be stored oneach weight sensor 110 a-i—see, e.g., FIG. 13 ). Since items 3924 a-imay be moved from their predefined locations over time (e.g., as peopleinteract with the items 3924 a-i), it may be beneficial to supplementitem assignment determinations that are based, at least in part, onweight sensor 110 a-i measurements (see, e.g., FIGS. 12-17 and 33A-37and corresponding description above) with the image-based itemassignment determinations described with respect to FIGS. 40-44 below.

FIG. 40 is a flow diagram 4000 illustrating an example operation of thetracking subsystem 3910 of FIG. 1 . The tracking subsystem 3910 mayexecute the instructions 4004, 4010, 4012, and 4016 described in FIG. 2(e.g., using one or more processors as described with respect to thedevice of FIG. 38 . The various instructions 4004, 4010, 4012, and 4016generally include any code, logic, and/or rules for implementing thecorresponding functions described below with respect to FIG. 40 .

An example operation of the tracking subsystem 3910 may begin with thereceipt of a proximity trigger 4002. The proximity trigger 4002 may beinitiated based on a proximity of the person 3902 to the rack 112. Forexample, top-view images 3908 captured by the top-view sensor 3904 shownin FIG. 1 may be used to initiate proximity trigger 4002. In particular,the tracking subsystem 3910 may detect, based on one or more top-viewimages 3908 that the person 3902 is within the threshold distance 3912of the rack 112. In some embodiments, the proximity trigger 4002 may bedetermined based on angled-view images 3918 received from theangled-view sensor 3914. For example, the proximity trigger 4002 may beinitiated upon determining, based on one or more angled-view images3918, that the person 3902 is within the threshold distance 3912 of therack 112 or that the person 3902 has entered the field-of-view 3916 ofthe angled-view sensor 3914. In some cases, it may be beneficial toinitiate the proximity trigger 4002 based on top-view images 3908 (e.g.,which may already be collected by the system 100 of FIG. 1 to performperson tracking) and begin collecting and analyzing angled-view images3918 following the proximity trigger 4002 in order to perform the itemassignment tasks described further below. This may improve overallefficiency by reserving the processing resources associated withcollecting, processing, and analyzing angled-view images 3918 until anitem assignment is likely to be needed after receipt of the proximitytrigger 4002.

a. Event Trigger and Item Localization

Following the proximity trigger 4002, the tracking subsystem 3910 maybegin to implement the item localization/event trigger instructions4004. The item localization/event trigger instructions 4004 generallyfacilitate the detection of an event associated with an item 3924 a-ibeing interacted with by the person 3902 (e.g., detecting vibrations onthe rack 112 using an accelerometer, detecting weight changes on weightsensor 110, detecting an item 3924 a-i being removed from or placed on ashelf 3920 a-c of the rack 112, etc.) and/or the identification of anapproximate location 4008 of the interaction (e.g., the wrist positions4102 and/or aggregated wrist position 4106 illustrated in FIG. 41 ,described below).

The item localization/event trigger instructions 4004 may cause thetracking subsystem 3910 to begin receiving angled-view images 3918 ofthe rack 112 (e.g., if such images 3918 are not already being received).The tracking subsystem 3910 may identify a portion of the angled-viewimages 3918 to analyze in order to determine whether a shelf-interactionevent has occurred (e.g., to determine whether the person 3902 likelyinteracted with an item 3924 a-i stored on the rack 112. This portion ofthe angled-view images 3918 may include image frames from a timeframeassociated with a possible person-shelf, or person-item, interaction.The portion of the angled-view images 3918 may be identified based on asignal from a weight sensor 110 a-i indicating a change in weight. Forinstance, a decrease in weight may indicate an item 3924 a-i may havebeen removed from the rack 112 and an increase in weight may indicate anitem 3924 a-i was placed on the rack 112. The time at which a change inweight occurs may be used to determine the timeframe associated with theinteraction. Detection of a change of weight is described in greaterdetail above with respect to FIGS. 12-17 and 33A-37 (e.g., see step 1502of FIG. 15 and corresponding description above). The tracking subsystem3910 may also or alternatively identify the portion of the angled-viewimages 3918 to use for item assignment by detecting the person 3902entering a zone adjacent to the rack 112 (e.g., as described withrespect to the “virtual curtain” of FIGS. 12-15 above and/or zone 3324described in greater detail above with respect to FIG. 33 ).

An example depiction of an angled-view image 3918 from one of theidentified frames is illustrated in FIG. 41 as image 4100. Image 4100includes the person 3902 and at least a portion of the rack 112. Thetracking subsystem 3910 uses pose estimation (e.g., as described withrespect to FIGS. 33C and 34 ) to determine pixel positions 4102 of, inone embodiment, a wrist of the person 3902 in each image 3918 from theidentified frames. In other embodiments, the pixel positions of otherrelevant parts of the body of a person 3902 (e.g., fingers, hand, elbow,forearm, etc.) may be used by the tracking subsystem 3910 in conjunctionwith pose estimation to perform the operations described below. Forexample, a skeleton 4104 may be determined using a pose estimationalgorithm (e.g., as described with respect to the determination ofskeletons 3302 e and 3302 f shown in FIG. 33C). A first wrist position4102 a on the skeleton 4104 is shown at the position of the person'swrist in image 4100 of FIG. 41 . FIG. 41 also shows a set of wrist pixelpositions 4102 determined during the remainder of the identified frames(e.g., during the rest of the timeframe determined to be associated withthe shelf interaction).

The tracking subsystem 3910 may then determine an aggregated wristposition 4106 based on the set of pixel positions 4102. For example, theaggregated wrist position 4106 may correspond to a maximum depth intothe rack 112 to which the person 3902 reached to possibly interact withan item 3924 a-i. This maximum depth may be determined, at last in part,based on the angle, or orientation, of the angled-view camera 3914relative to the rack 112. For instance, in the example of FIG. 41 wherethe angled-view sensor 3914 provides a view over the right shoulder ofthe person 3902 relative to the rack 112, the aggregated wrist position4106 may be determined as the right-most wrist position 4102. If anangled-view sensor 394 provides a different view, a different approachmay be used to determine the aggregated wrist position 4106 asappropriate. If the angled-view sensor 3914 provides depth information(e.g., if the angled view sensor 3914 includes a depth sensor), thisdepth information may be used to determine the aggregated wrist position4106.

Referring to FIGS. 40 and 41 , the aggregated wrist position 4106 may beused to determine if an event trigger 4006 should be initiated. Forexample, the tracking subsystem 3910 may determine whether theaggregated wrist position 4106 corresponds to a position on a shelf 3920a-c of the rack 112. This may be achieved by comparing the aggregatedwrist position 4106 to a set of one or more predefined shelf positions(e.g., determined based at least in part on the shelf markers 3922 a-c,described above). Based on this comparison, the tracking subsystem 3910may determine whether the aggregated wrist position 4106 is within athreshold distance of at least one of the shelves 3920 a-c of the rack112 or to a predefined location of the item 3924 a-i on the shelf 3920a-c. If the aggregated wrist position 4106 is within a thresholddistance of a shelf 3920 a-c (e.g., or of a predefined position of anitem 3924 a-i stored on a shelf 3920 a-c), the event trigger 4006 ofFIG. 40 may be initiated (e.g., provided for data handling andintegration, as illustrated in FIG. 40 ). Thus, the event trigger 4006indicates that a shelf-interaction event has likely occurred, such thatfurther tasks of item identification and/or action type identificationare appropriate.

The item localization/event trigger instructions 4004 may furtherdetermine an event location 4008. The event location 4008 may include animage region that is associated with the detected shelf 3920 a-cinteraction, as illustrated by the dashed-line region in FIG. 41 . Forexample, the event location 4008 may include the aggregated wristposition 4106 and a region of the image 4100 surrounding the aggregatedwrist position 4106. As described further below, the tracking subsystem3910 may use the event location 4008 to facilitate improved itemidentification (e.g., using the item identification instructions 4012)and/or improved activity recognition (e.g., using the activityrecognition instructions 4016), as described further below. Althoughevent location 4008 illustrated in FIG. 41 is illustrated as arectangle, it should be understood that any appropriate size and/orshape of event location 4008 may be used in accordance with the sizeand/or shape of the particular body part of the person 3902 whose pixelpositions are used by tracking subsystem 3910.

As an example, the tracking subsystem 3910 may determine at least oneimage 3918 associated with the person 3902 removing an item 3924 a-ifrom the rack 112. The tracking subsystem 3910 may determine aregion-of-interest (e.g., region-of interest 4202 described with respectto FIG. 42 below) within this image 4100 based on the aggregated wristlocation 4106 and/or the event location 4008 and use an objectrecognition algorithm to identify the item 3924 a-i within theregion-of-interest (see, e.g., descriptions of the implementation of theitem identification instructions 4012 and FIGS. 42 and 44 below).Although region-of-interest 4202 illustrated in FIG. 41 is illustratedas a circle, it should be understood that any appropriate size and/orshape of region-of-interest 4202 may be used in accordance with the sizeand/or shape of the particular body part of the person 3902 whose pixelpositions are used by tracking subsystem 3910.

As another example, the tracking subsystem 3910 may determine, based onthe aggregated wrist position 4106 and/or the event location region4008, candidate items that may have been removed from the rack by theperson 4008. For example, the candidate items may include a subset ofall the items 3924 a-i stored on the shelves 3920 a-c of the rack 112that have predefined locations (e.g., see FIG. 13 ) that are within theregion defined by the event location 4008 and/or are within a thresholddistance from the aggregated wrist position 4106. Identification ofthese candidate items may narrow the search space for identifying theitem 3924 a-i with which the person 3902 interacted, thereby improvingoverall efficiency.

Referring again to FIG. 40 , further use of the event trigger 4006and/or event location 4008 for item assignment may be coordinated usingthe data feed handling and integration instructions 4010. The data feedhandling and integration instructions 4010 generally include code,logic, and/or rules for communicating the event trigger 4006 and/orevent location 4008 for use by the item identification instructions 4012and/or activity recognition instructions 4016, as illustrated in FIG. 40. Data feed handling and integration instructions 4010 may help ensurethat the correct information is appropriately routed to perform furtherfunctions of the tracking subsystem 3910 (e.g., tasks performed by theother instructions 4012 and 4016).

As is described further below, the data feed handling and integrationinstructions 4010 also integrate the information received from each ofthe item localization/event trigger instructions 4004, the itemidentification instructions 4012, and the activity recognitioninstructions 4016 to determine an appropriate item assignment 4020. Theitem assignment 4020 generally refers to an indication of an item 3924a-i interacted with by the person 3920 (e.g., based on the itemidentifier 3912 determined by the item identification instructions 4012)and an indication of whether the item 3924 a-i was removed from the rack112 or placed on the rack 112 (e.g., based on the item removed orreplaced 4018 determined by the activity recognition instructions 4016).The item assignment 4020 is used to appropriately update the person'sdigital shopping cart 3926 by appropriately adding or removing items3924 a-i. For example, the digital shopping cart 3926 may be updated toinclude the appropriate quantity 3932 for the item identifier 3930 ofthe item 3924 a-i.

b. Object Detection Based on Wrist-Area Region-of-Interest

Still referring to FIG. 40 , the item identification instructions 4012may receive the event trigger 4006 and/or event location 4008. Followingreceipt of the event trigger 4006, the tracking subsystem 3910 maydetermine one or more images 3918 (e.g., of the overall feed ofangled-view images 3918) that are associated with the event trigger4006. These images 3918 may be all or a portion of the images 3918 inwhich wrist positions 4102 (or other body parts, as appropriate) weredetermined, as described above with respect to FIG. 41 . FIG. 42illustrates an example representation 4200 of an identifiedevent-related image 3918 in which the person 3902 is interacting with afirst item 3924 a on the top shelf 3920 a of the rack 112. In at leastthis image 4200, the tracking subsystem 3910 uses pose estimation todetermine the wrist position 4102 (e.g., a pixel position of the wrist)of the person 3902. This wrist position 4202 may already have beendetermined by the item localization/event trigger instructions 4004 andincluded in the event location 4008 information, or the trackingsubsystem 3910 may determine this wrist position 4102 (e.g., bydetermining the skeleton 4104, as described with respect to FIG. 41above).

Following determination of the wrist position 4102, the trackingsubsystem 3910 then determines, in the image 4100, a region-of-interest4202 within the image 4200 based on the wrist position 4102. Theregion-of-interest 4202 illustrated in FIG. 42 is a circularregion-of-interest. However, the region-of-interest 4202 can be anyshape (e.g., a square, rectangle, or the like). The region-of-interest4202 includes a subset of the entire image 4200, such that itemidentification may be performed more efficiently in theregion-of-interest 4202 than would be possible using the entire image4200. The region-of-interest 4202 has a size 4204 that is sufficient tocapture a substantial portion of the item 3924 a to identify the item3924 a using an image recognition algorithm. For the exampleregion-of-interest 4202 of FIG. 42 , the size 4204 of theregion-of-interest corresponds to a radius. For a region-of-interest4202 with a different shape, a different size 4204 parameter maycharacterize the region-of-interest 4202 (e.g., a width for a square, alength and width for a rectangle, etc.). The size 4204 of theregion-of-interest 4202 may be a predetermined value (e.g.,corresponding to a predefined number of pixels in the image 4200 or apredefined physical length in the space 102 of FIG. 1 ). In someembodiments, the region-of-interest 4202 has a size that is based onfeatures of the person 3902, such as the shoulder width 4206 of theperson 3902, the arm length 4208 of the person 3902, the height 4210 ofthe person 3902, and/or value derived from one or more of these or otherfeatures. As an example, the size 4204 of the region-of-interest 4202may be proportional to a ratio of the shoulder width 4206 of the person3902 to the arm length 4208 of the person 3902.

The tracking subsystem 3910 may identify the item 3924 a by determiningan item identifier 4014 (e.g., the identifier 3930 of FIG. 39 ) for theitem 3924 a located within the region-of-interest 4202. Generally, theimage of the item 3924 a within the region-of-interest 4202 may becompared to images of candidate items. The candidate items may includeall items offered for sale in the space 102, items 3924 a-i stored onthe rack 112, or a subset of the items 3924 a-i that are stored on therack 112, as described further below. In some cases, for each candidateitem, a probability is determined that the candidate item is the item3924 a. The probability may be determined, at least in part, based on acomparison of a predefined position associated with the candidate items(e.g., the predefined location of items 3924 a-i on the rack 112—seeFIG. 13 ) to the wrist position 4102 and/or the aggregated wristposition 4106. In some embodiments, the probability for each candidateitem may be determined using an object detection algorithm 3934. Theobject detection algorithm 3934 may employ a neural network or a methodof machine learning (e.g. a machine learning model). The objectdetection algorithm 3934 may be trained using the range of items 3924a-i expected to be stored on the rack 112. For example, the objectdetection algorithm 3934 may be trained using previously obtained imagesof products offered for sale on the rack 112. Generally, an itemidentifier 4014 for the candidate item with the largest probabilityvalue (e.g., and that is at least a threshold value) is assigned to theitem 3924 a. In some embodiments, the tracking subsystem 3910 mayidentify the item 3924 a using feature-based techniques, contrastiveloss-based neural networks, or any other suitable type of technique foridentifying the item 3924 a.

Since decreasing the number of candidate items can facilitate more rapiditem identification, the event location 4008 or other item positioninformation may be used to narrow the search space for correctlyidentifying the item 3924 a. For example, prior to identifying the item3924 a with which the person 3902 interacted, the tracking subsystem3910 may determine candidate items that are known to be located near theregion-of-interest 4202, near the aggregated wrist position 4106described above with respect to FIG. 41 , and/or within the regiondefined by the event location 4008 (see example dashed-line region inFIG. 41 ). The candidate items include at least the item 3924 a and mayalso include and one or more other items 3924 b-i stored on the rack112. For example, the other candidate items may have known positions(see FIG. 13 ) at adjacent positions in the rack 112 to the item 3924 a.For example, the candidate items may include items with predefinedlocations at the position of item 3924 a, and the adjacent positions ofitems 3924 b,d,e.

c. Detection of Object Removal and Replacement

After the tracking subsystem 3910 has determined that the person 3902has interacted with the item 3924 a, further actions may be needed todetermine whether the item 3924 a was removed from the rack 112 orplaced on the rack 112. For instance, the tracking subsystem 3910 maynot have reliable information about whether the item 3924 a was takenfrom the rack 112 or placed on the rack 112. In cases where the item3924 a was on a weight sensor 110 a, a change of weight associated withthe interaction detected by the tracking subsystem 3910 may be used todetermine whether the item 3924 a was removed or placed on the rack 112.For example, a decrease in weight at the weight sensor 3910 a mayindicate the item 3924 a was removed from the rack, while an increase inweight may indicate the item 3924 a was placed on the rack 112. In caseswhere there is no weight sensor 3910 a or when a weight change isinsufficient to provide reliable information about whether the item 3924a was removed from or placed on the rack 112 (e.g., if the magnitude ofthe change of weight does not correspond to an expected weight changefor the item 3924 a), the tracking subsystem 3910 may track the item3924 a through the space 102 after it exits the rack 112 (see FIGS.36A-B and 37). However, as described above, it may be advantageous toavoid unnecessary further item tracking in order to more efficiently usethe processing and imaging resources of the tracking subsystem 3910. Theactivity recognition instructions 4016, described below, facilitate thereliable determination of whether the item 3924 a was removed from orplaced on the rack 112 without requiring weight sensors 110 a-i orsubsequent person tracking and item re-evaluation as the person 3902continues to move about the space 102 (see FIGS. 36A-B and 37). Theactivity recognition instructions 4016 also facilitate more rapididentification of whether the item 3924 a was removed from or placed onthe rack 112 than may have been possible previously, thereby reducingdelays in item assignment which may otherwise result in a relativelypoor user experience.

Referring again to FIG. 40 , the activity recognition instructions 4016may receive the item identifier 4014 and the event location 4008 and usethis information, at least in part, to determine whether the item 3924 awas removed from or placed on the rack 112. For example, the trackingsubsystem 3910 may identify a time interval associated with theinteraction between the person 3902 and the item 3924 a. For example,the tracking subsystem 3910 may identify a first image 3918 (e.g., image4302 a of FIG. 43 ) corresponding to a first time before the person 3902interacted with the item 3924 a and a second image (e.g., image 4302 bof FIG. 43 ) corresponding to a second time after the person 3902interacted with the item 3924 a (see example of FIG. 43 , describedbelow). Based on a comparison of the first image 3918 to the secondimage 3918, the tracking subsystem 3910 determines an indication 4018 ofwhether the item 3924 a was removed from the rack 112 or the item 3924 awas placed on the rack 112.

The data feed handling and integration instructions may use theindication 4018 of whether the item 3924 a was removed or replaced onthe rack 112 to determine the item assignment 4020. For example, if itis determined that the item 3924 a was removed from the rack 112, theitem 3924 a may be assigned to the person 3902 (e.g., to the digitalshopping cart 3926 of the person 3902). Otherwise, if it is determinedthat the item was placed on the rack 112, the item 3924 a may beunassigned from the person 3902. For instance, if the item 3924 a wasalready present in the person's digital shopping cart 3926, then theitem 3924 a may be removed from the digital shopping cart 3926.

Detection of Object Removal and Replacement from a Shelf

FIG. 43 is an example flow diagram 4300 illustrating an example approachemployed by the tracking subsystem 3910 to determine whether the item3924 a was placed on the rack 112 or removed from the rack 112 using theactivity recognition instructions 4016. As described above, the trackingsubsystem 3910 may determine a first image 4302 a corresponding to afirst time 4304 a before the person 3902 interacted with the item 3924a. For example, the tracking subsystem 3910 may identify an interactiontime associated with the person 3902 interacting with the rack 112, theshelf 3920 a, and/or the item 3924 a. For instance, this interactiontime may be determined as a time at which the item 3924 a was identified(e.g., in the region-of-interest 4202 illustrated in FIG. 42 , describedabove). The first time 4304 a may be a time that is before thisinteraction time. For instance, the first time 4304 a may be theinteraction time minus a predefined time interval (e.g., of several totens of seconds). The first image 4302 a is an angled-view image 3918 ator near the first time 4304 a (e.g., with a timestamp correspondingapproximately to the first time 4304 a).

The tracking subsystem 3910 also determines a second image 4302 bcorresponding to a second time 4304 b after the person 3902 interactedwith the item 3924 a. For example, the tracking subsystem 3910 maydetermine the second time 4304 b based on the interaction timeassociated with the person 3902 interacting with the rack 112, the shelf3920 a, and/or the item 3924 a, described above. The second time 4304 bmay be a time that is after the interaction time. For example, thesecond time 4304 b may be the interaction time plus a predefined timeinterval (e.g., of several to tens of seconds). The second image 4302 bis an angled-view image 3918 at or near the second time 4304 b (e.g.,with a timestamp corresponding approximately to the second time 4304 b).

The tracking subsystem 3910 then compares the first and second images4302 a,b to determine whether the item 3924 a was added to or removedfrom the rack 112. The tracking subsystem 3910 may first determine aportion 4306 a of the first image 4302 a and a portion 4306 b of thesecond image 4302 b that each correspond to a region around the item3924 a in the corresponding image 4302 a,b. For example, the portions4306 a,b may each correspond to a region-of-interest (e.g.,region-of-interest 4202 of FIG. 42 ) associated with the interactionbetween the person 3902 and the object 3924 a. While the portions 4306a,b of images 4302 a,b are shown as a rectangular region in the image,the portions 4306 a,b may generally be any appropriate shape (e.g., acircle as in the region-of-interest 4202 shown in FIG. 42 , a square, orthe like). Comparing image portion 4306 a to image portion 4306 b isgenerally less computationally expensive and may be more reliable thancomparing the entire images 4304 a,b. The tracking subsystem 3910provides the portion 4306 a of the first image 4302 a and the portion4306 b of the second image 4302 b to a neural network 4308 trained todetermine a probability 4310, 4314 corresponding to whether the item3924 a has been added or removed from the rack 112 based on a comparisonof the two input images 4306 a,b. For example, the neural network 4308may be a residual neural network. The neural network 4308 may be trainedusing previously obtained images of the item 3924 a and/or similaritems. If a high probability 4310 is determined (e.g., a probability4310 that is greater than or equal to a threshold value), the trackingsubsystem 3910 generally determines that the item 3924 a was returned4312, or added to, the rack 112. If a low probability 4314 is determined(e.g., a probability 4314 that is less than the threshold value), thetracking subsystem 3910 generally determines that the item 3924 a wasremoved 4316 from the rack 112. In the example of FIG. 43 , the item3924 a was removed. The returned 4312 or removed 4316 determination isprovided as the indication 4018 shown in FIG. 40 to the data feedhandling and integration instructions 4010 to complete the itemassignment 4020, as described above.

Example Method of Item Assignment

FIG. 44 illustrates a method 4400 of operating the tracking subsystem3910 to perform functions of the item localization/event triggerinstructions 4004, item identification instructions 4012, activityrecognition instructions 4016, and the data feed handling andintegration instructions 4010, described above. The method 4400 maybegin at step 4402 where a proximity trigger 4002 is detected. Asdescribed above with respect to FIG. 40 , the proximity trigger 4002 maybe initiated based on the proximity of the person 3902 to the rack 3912.For example, the tracking subsystem 3910 may detect, based on one ormore top-view images 3908 that the person 3902 is within a thresholddistance 3912 of the rack 112. In some embodiments, the proximitytrigger 4002 may be determined based on angled-view images 3918 receivedfrom the angled-view sensor 3914. For example, the proximity trigger4002 may be initiated upon determining, based on one or more angled-viewimages 3918, that the person 3902 is within the threshold distance 3912of the rack 112 or that the person 3902 has entered the field-of-view3916 of the angled-view sensor 3914.

At step 4404, the wrist position 4102 (e.g., a pixel positioncorresponding to the location of the wrist of the person 3902 in images3918) of the person 3902 is tracked. For example, the tracking subsystem3910 may perform pose estimation (e.g., as described with respect toFIGS. 33C, 34, 40, 41, and 42 ) to determine pixel positions 4102 of awrist of the person 3902 in each image 3918. For example, a skeleton4104 may be determined using a pose estimation algorithm (e.g., asdescribed with respect to the determination of skeletons 3302 e and 3302f shown in FIG. 33C and skeleton 4104 of FIGS. 41 and 42 ). Examplewrist positions 4102 are shown in FIG. 41 .

At step 4406, an aggregated wrist position 4106 is determined. Theaggregated wrist position 4106 may be determined based on the set ofwrist positions 4102 determined at step 4404. For example, theaggregated wrist position 4106 may correspond to a maximum depth intothe rack 112 to which the person 3902 reached to interact with an item3924 a-i. This maximum depth may be determined, at last in part, basedon the angle of the angled-view camera 3914 relative to the rack 112.For instance, in the example of FIG. 41 where the angled-view sensor3914 provides a view over the right shoulder of the person 3902 relativeto the rack 112, the aggregated wrist position 4106 may be determined asthe right-most wrist position 4102. If an angled-view sensor 3914provides a different view, a different approach may be used to determinethe aggregated wrist position 4106, as appropriate. If the angled-viewsensor 3914 provides depth information (e.g., if the angled view sensor3914 includes a depth sensor), this depth information may be used todetermine the aggregated wrist position 4106.

At step 4408, the tracking subsystem 3910 determines if an event (e.g.,a person-rack interaction event) is detected. For example, the trackingsubsystem 3910 may determine whether the aggregated wrist position 4106is within a threshold distance of a predefined location of an item 3924a-i. If the aggregated wrist position 4106 is within the thresholddistance of a predefined position of an item 3924 a-i, an event may bedetected. In some embodiments, the tracking subsystem 3910 may determineif an event is detected based on a change of weight indicated by aweight sensor 3910 a-i. If a change of weight is detected, an event maybe detected. If an event is not detected, the tracking subsystem 3910may return to wait for another proximity trigger 4002 to be detected atstep 4402. If an event is detected, the tracking subsystem 3910generally proceeds to step 4410.

At step 4410, candidate items 3924 a-i may be determined based on theaggregated wrist position 4106. For example, the candidate items mayinclude a subset of all the items 3924 a-i stored on the shelves 3920a-c of the rack 112 that have predefined locations (e.g., see FIG. 13 )that are within the region defined by the event location 4008 and/or arewithin a threshold distance from the aggregated wrist position 4106.Identification of these candidate items may narrow the search space foridentifying the item 3924 a-i with which the person 3902 interacted atsteps 4412 and/or 4418 (described below), thereby improving overallefficiency of item 3924 a-i assignment.

At step 4412, the tracking subsystem 3910 may determine the identity ofthe interacted with item 3924 a-i (e.g., item 3924 a of the examples ofimages FIGS. 41 and 42 ) based on the aggregated wrist position 4106.For example, the tracking subsystem 3910 may determine that theinteracted-with item 3924 a-i is the item 3924 a-i with a predefinedlocation on the rack 112 that is nearest the aggregated wrist position4106. In some cases, the tracking subsystem 3910 may determine aprobability that the person interacted with each of the candidate itemsdetermined at step 4410. For example, a probability may be determinedfor each candidate item, where the probability for a given candidateitem is increased when the distance between the aggregated wristposition 4106 and the predefined position of the candidate item isdecreased.

At step 4414, the tracking subsystem 3910 determines if further item3924 a-i identification is appropriate. For example, the trackingsubsystem 3910 may determine whether the identification at previous step4412 satisfies certain reliability criteria. For instance, if thehighest probability determined for candidate items is less than athreshold value, the tracking subsystem 3910 may determine that furtheridentification is needed. As another example, if multiple candidateitems were likely interacted with by the person 3902 (e.g., ifprobabilities for multiple candidate items were greater than a thresholdvalue), then further identification may need to be performed. If achange in weight was received when the person 3902 interacted with anitem 3924 a-i, the tracking subsystem 3910 may compare a predefinedweight for each candidate item to the change in weight. If the change inweight matches one of the predefined weights, then no furtheridentification may be needed. However, if the change in weight does notmatch one of the predefined weights of the candidate items, then furtheridentification may be needed. If further identification is needed, thetracking subsystem 3910 proceeds to step 4416. However, if no furtheridentification is needed, the tracking subsystem 3910 may proceed tostep 4420.

At steps 4416 and 4418, the tracking subsystem 3910 performs objectrecognition-based item identification. For example, at step 4416, thetracking subsystem 3910 may determine a region-of-interest 4202associated with the person-rack interaction. As described above, theregion-of-interest 4202 includes a subset of an image 3918, such thatobject recognition may be performed more efficiently within this subsetof the entire image 3918. As described above, the region-of-interest4202 has a size 4204 that is sufficient to capture a substantial portionof the item 3924 a-i to identify the item 3924 a-i using an imagerecognition algorithm at step 4418. The size 4204 of theregion-of-interest 4202 may be a predetermined value (e.g.,corresponding to a predefined number of pixels in the image 3918 or apredefined physical length in the space 102). In some embodiments, theregion-of-interest 4202 has a size that is based on features of theperson 3902, such as the shoulder width 4206 of the person 3902, the armlength 4208 of the person 3902, the height 4210 of the person 3902,and/or value derived from one or more of these or other features.

At step 4418, the tracking subsystem 3910 identifies the item 3924 a-iwithin the region-of-interest 4202 using an object recognitionalgorithm. For example, the image of the item 3924 a-i within theregion-of-interest 4202 may be compared to images of candidate itemsdetermined at step 4410. In some cases, for each candidate item, aprobability is determined that the candidate item is the item 3924 a-i.The probability for each candidate item may be determined using anobject detection algorithm 3934. The object detection algorithm 3934 mayemploy a neural network or a method of machine learning (e.g. a machinelearning model). The object detection algorithm 3934 may be trainedusing the range of items 3924 a-i expected to be presented on the rack112. For example, the object detection algorithm 3934 may be trainedusing previously obtained images of products offered for sale on therack 112. Generally, an item identifier 4014 for the candidate item withthe largest probability value (e.g., that is at least a threshold value)is assigned to the item 3924 a-i.

At steps 4420 4422, the tracking subsystem 3910 may determine whetherthe identified item 3924 a-i was removed from or placed on the rack 112.For example, at step 4420, the tracking subsystem 3910 may determine afirst image 4302 a before the interaction between the person 3902 andthe item 3924 a-i and a second image 4302 b after the interactionbetween the person 3902 and the item 3924 a-i. At step 4422, thetracking subsystem 3910 determines if the item 3924 a-i was removed fromor placed on the rack 112. This determination may be based on acomparison of the first and second images 4302 a,b, as described abovewith respect to FIGS. 40 and 43 . If a change in weight was receivedfrom a weight sensor 110 a-i for the person-item interaction (e.g., ifthe rack includes a weight sensor 110 a-i for one or more of the items3924 a-i), the change in weight may be used, at least in part, todetermine if the item 3924 a-i was removed from or placed on the rack112. For example, if the weight on a sensor 110 a-i decreases, then thetracking subsystem 3910 may determine that the item 3924 a-i was removedfrom the rack 112. If the item 3924 a-i was removed from the rack 112,the tracking subsystem 3910 proceeds to step 4424. Otherwise, if theitem 3924 a-i was placed on the rack 112, the tracking subsystem 3910proceeds to step 4426.

At step 4424, the tracking subsystem 3910 assigns the item 3924 a-i tothe person 3902. For example, the digital shopping cart 3926 may beupdated to include the appropriate quantity 3932 of the item 3924 a-iwith the item identifier 3930 determined at step 4412 or 4418. At step4426, the tracking subsystem 3910 determines if the item 3924 a-i wasalready assigned to the person 3902 (e.g., if the digital shopping cart3926 includes an entry for the item identifier 3930 determined at step4412 or 4418). If the item 3924 a-i was already assigned to the person3902, then the tracking subsystem 3910 proceeds to step 4428 tounassigned the item 3924 a-i from the person 3902. For example, thetracking subsystem 3910 may remove a unit of the item 3924 a-i from thedigital shopping cart 3926 of the person 3902 (e.g., by decreasing thequantity 3932 in the digital shopping cart 3926). If the item 3924 a-iwas not already assigned, the tracking subsystem 3910 proceeds to step4420 and does not assign the item 3924 a-i to the person 3902 (e.g.,because the person 3902 may have touched and/or moved the item 3924 a-ion the rack 112 without necessarily picking up the item 3924 a-i).

Self-Serve Beverage Assignment

In some cases, the space 102 (see FIG. 1 ) may include one or moreself-serve beverage machines, which are configured to be operated by aperson to dispense a beverage into a cup. Examples of beverage machinesinclude coffee machines, soda fountains, and the like. While the varioussystems, devices, and processes described above generally facilitate thereliable assignment of items selected from a rack 112 or other locationin the space 102 to the correct person, further actions and/ordeterminations may be needed to appropriately assign self-servebeverages to the correct person. Previous technology generally relies ona cashier identifying a beverage and/or a person self-identifying aselected beverage. Thus, previous technology generally lacks the abilityto automatically assign a self-serve beverage to a person who wishes topurchase the beverage.

This disclosure overcomes these and other technical problems of previoustechnology by facilitating the identification of self-serve beveragesand the assignment of self-serve beverages to the correct person usingcaptured video/images of interactions between people and self-servebeverage machines. This allows the automatic assignment of self-servebeverages to a person's digital shopping cart without humanintervention. Thus, a person may be assigned and ultimately charged fora beverage without ever interacting with a cashier, a self-checkoutdevice, or other application. An example of a system 4500 for detectingand assigning beverages is illustrated in FIG. 45 . In some cases,beverage assignment may be performed primarily using image analysis, asillustrated in the example method of FIG. 46 . In other embodiments, thebeverage machine or an associated sensor may provide an indication of atype and/or quantity of a beverage that is dispensed, and thisinformation may be used in combination with image analysis toefficiently and reliably assign self-serve beverages to the correctpeople, as described in the example method of FIG. 47 .

FIG. 45 illustrates an example system 4500 for self-serve beverageassignment. The system 4500 includes a beverage machine 4504, anangled-view sensor 4520, and a beverage assignment subsystem 4530. Thebeverage assignment system 4500 generally facilitates the automaticassignment of a beverage 4540 to a person 4506 (e.g., by including anidentifier of the beverage 4540 in a digital shopping cart 4538associated with the person 4502). The beverage assignment system 4500may be included in the system 100, described above, or used to assignbeverages to people moving about the space 102, described above (seeFIG. 1 ).

The beverage machine 4504 is generally any device operable to dispense abeverage 4540 to a person 4502. For example, the beverage machine 4504may be a coffee pot resting on a hot plate, a manual coffee dispenser(as illustrated in the example of FIG. 45 ), an automatic coffeemachine, a soda fountain, or the like. A beverage machine 4504 mayinclude one or more receptacles (not pictured for clarity andconciseness) that hold a prepared beverage 4540 (e.g., coffee) and amechanism 4504 which can be operated to release the prepared beverage4540 into a cup 4508. In the example, of FIG. 45 , the dispensingmechanism 4506 is a spout (e.g., a manually actuated valve that controlsthe release of the beverage 4540). In some embodiments, a beveragemachine 4504 includes a flow meter 4510, which is configured to detect aflow of beverage 4540 (e.g., out of the dispensing mechanism 4506) andprovide a flow trigger 4512 to the beverage assignment subsystem 4530,described further below. For example, a conventional coffee dispensermay be retrofitted with a flow meter 4510, such that an electronic flowtrigger 4510 can facilitate the detection of interactions with thebeverage machine 4504. As described further below, the flow trigger mayinclude information about the time a beverage 4540 is dispensed alongwith the amount and/or type of beverage 4540 dispensed. The flow trigger4512 may be used to select images 4528 from appropriate times forassigning the beverage to the correct person 4502.

In some cases, a beverage machine 4504 may be configured to prepare abeverage 4540 based on a user's selection. For instance, such a beveragemachine 4504 may include a user interface (e.g., buttons, a touchscreen,and/or the like) for selecting a beverage type and size along with oneor more receptacles that hold beverage precursors (e.g., water, coffeebeans, liquid and/or powdered creamer, sweetener(s), flavoring syrup(s),and the like). Such a beverage machine 4504 may include a devicecomputer 4514, which may include one or more application programminginterfaces (APIs) 4516, which facilitate communication of informationabout the usage of the beverage machine 4504 to the beverage assignmentsubsystem 4530. For example, the APIs 4516 may provide a device trigger4518 to the beverage assignment subsystem 4530. The device trigger 4518may include information about the time at which a beverage 4540 wasdispensed, the type of beverage dispensed, and/or the amount of beverage4540 dispensed. Similar to the flow trigger 4512, the device trigger4518 may also be used to select images 4528 from appropriate times forassigning the beverage to the correct person 4502. While the example ofFIG. 45 shows a manually operated beverage machine 4504 that dispenses abeverage 4540 into a cup 4508 placed below the dispensing mechanism4506, this disclosure contemplates the beverage machine 4504 being anymachine that is operated by a person 4502 to dispense a beverage 4540.

The angled-view sensor 4520 is configured to generate angled-view images4528 (e.g., color and/or depth images) of at least a portion of thespace 102. The angled-view sensor 4520 generates angled-view images 4528for a field-of-view 4522 which includes at least a zone 4524encompassing the dispensing mechanism 4506 of the beverage machine 4504and a zone 4526 which encompasses a region where a cup 4508 is placed toreceive dispensed beverage 4540. The angled-view sensor 4520 may includeone or more sensors, such as a color camera, a depth camera, an infraredsensor, and/or the like. The angled-view sensor 4520 may be the same asor similar to the angled-view sensor 108 b of FIG. 22 or the angled-viewsensor 3914 of FIG. 39 . In some embodiments, the beverage machine 4504includes at least one visible marker 4542 (e.g., the same as or similarto markers 3922 a-c of FIG. 39 ) positioned and configured to identify aposition of one or both of the first zone 4524 and the second zone 4526.The beverage assignment subsystem 4530 may detect the marker(s) 4542 andautomatically determine, based on the detected marker 4542, an extentthe first zone 4524 and/or the second zone 4526.

The angled-view images 4528 are provided to the beverage assignmentsubsystem 4530 (e.g., the server 106 and/or client(s) 105 describedabove) and used to detect the person 4502 dispensing beverage 4540 intoa cup 4508, identify the dispensed beverage 4540, and assign thebeverage 4540 to the person 4502. For example, the beverage assignmentsubsystem 4530 may update the digital shopping cart 4538 associated withthe person 4502 to include an identification of the beverage 4540 (e.g.,a type and amount of the beverage 4540). The digital shopping cart 4538may be the same or similar to the digital shopping carts (e.g., digitalcart 1410 and/or digital shopping cart 3926) described above withrespect to FIGS. 12-18 and 39-44 .

The beverage assignment subsystem 4530 may determine a beverageassignment 4536 using interaction detection 4532 and pose estimation4534, as described in greater detail below with respect to FIGS. 46 and47 . In some cases, interaction detection 4532 involves the detection,based on angled-view images 4528, of an interaction between the person4502 and the beverage machine 4504, as described in greater detail withrespect to FIG. 46 below. For instance, an interaction may be detectedif the person 4502 (e.g., the hand, wrist, or other relevant body partof the person 4502) enters both the first zone 4524 associated withoperating the dispensing mechanism 4506 of the beverage machine 4504 andthe second zone 4526 associated with placing and removing a cup 4508 toreceive beverage 4540. In some cases, pose estimation 4534 may beperformed to provide further verification that beverage 4540 wasdispensed from the beverage machine 4504 (e.g., based on the location ofa wrist or hand of the person 4502 determined via pose estimation 4534).Beverage assignment 4536 is performed based on the characteristics ofthe detected interaction and/or the determined pose. For instance, abeverage 4540 may be added to the digital shopping cart of the person4502, if the beverage assignment subsystem 4530 determines that the handof the person 4502 entered both zones 4524 and 4526 and that the cup4508 remained in the second zone 4526 for at least a threshold time(e.g., such that the cup 4508 was in the zone 4526 a sufficient amountof time for the beverage 4540 to be dispensed). Further details ofbeverage assignment based on angled-view images 4528 are provided belowwith respect to FIG. 46 .

In other cases, such as the example illustrated in FIG. 47 , interactiondetection 4532 involves receipt of a trigger 4512 and/or 4518, whichindicates that beverage 4540 is dispensed. For instance, an interactionmay be detected if a trigger 4512 and/or 4518 is received. In somecases, pose estimation 4534 may be performed, using angled-view images4528, to provide further verification that the same person 4502dispensed the beverage 4540 and removed the beverage 4540 (e.g., the cup4508) from the zone 4526. The beverage 4540 may be added to the digitalshopping cart 4538 of the person 4502, if the same person 4502 dispensedthe beverage 4540 at an initial time (e.g., reached into zone 4526 toplace the cup 4508 to receive the beverage) and removed the cup 4508from the zone 4526 at a later time. Further details of beverageassignment based on a trigger 4512 and/or 4518 and angled-view images4528 are provided below with respect to FIG. 47 .

a. Image-Based Detection and Assignment

FIG. 46 illustrates a method 4600 of operating the system 4500 of FIG.45 to assign a beverage 4540 to a person using angled-view images 4528captured by the angle-view sensor 4520. The method 4600 may begin atstep 4602 where an image feed comprising the angled-view images 4528 isreceived by the beverage assignment subsystem 4530. In some embodiments,the angled-view images 4528 may be received after the person 4502 iswithin a threshold distance of the beverage machine 4502. For instance,top-view images captured by sensors 108 within the space 102 (see FIG. 1) may be used to determine when a proximity trigger (e.g., the same asor similar to proximity trigger 4002 of FIG. 40 ) should cause thebeverage assignment subsystem 4530 to begin receiving angled-view images4528. As described above with respect to FIG. 45 , the angled viewimages 4528 are from a field-of-view 4522 that encompasses at least aportion of the beverage machine 4504, including the first zone 4524associated with operating the dispensing mechanism 4506 of the beveragemachine 4504 and the second zone 4526 in which the cup 4508 is placed toreceive the beverage 4540 from the beverage machine 4504.

At step 4604, the beverage assignment subsystem 4530 detects, based onthe received angled-view images 4528 an event associated with an objectentering one or both of the first zone 4524 and the second zone 4526. Insome embodiments, the beverage assignment subsystem 4530 may identify asubset of the images 4528 that are associated with the detected eventand which should be analyzed in subsequent steps to determine thebeverage assignment 4536 (see FIG. 45 ). For instance, as part of or inresponse to detecting the event at step 4604, the beverage assignmentsubsystem 4530 may identify a first set of one or more images 4528associated with the start of the detected event. The first set of images4528 may include images 4528 from a first predefined time before thedetected event until a second predefined time period after the detectedevent. A second set of images 4528 may also be determined following thedetermination of the removal of the cup 4508 from the zone 4526 at step4612, described below. The second set of images 4528 include images 4528from a predefined time before the cup 4508 is removed from the secondzone 4526 until a predefined time after the cup 4508 is removed. Imageanalysis tasks may be performed more efficiently in these sets of images4528, rather than evaluating every image 4528 received by the beverageassignment sub system 4530.

At step 4606, the beverage assignment subsystem 4530 may perform poseestimation (e.g., using any appropriate pose estimation algorithm) todetermine a pose of the person 4502. For example, a skeleton may bedetermined using a pose estimation algorithm (e.g., as described abovewith respect to the determination of skeletons 3302 e and 3302 f shownin FIG. 33C and skeleton 4104 shown in FIGS. 41 and 42 ). Informationfrom pose estimation may inform determinations in subsequent steps 4608,4610, 4612, and/or 4614, as described further below.

At step 4608, the beverage assignment subsystem 4530 may determine anidentity of the person 4502 interacting with the beverage machine 4504.For example, the person 4502 may be identified based on features and/ordescriptors, such as height, hair color, clothing properties, and/or thelike of the person 4502 (see, e.g., FIGS. 27-32 and correspondingdescription above). In some embodiments, features may be extracted froma skeleton determined by pose estimation at step 4606. For instance, ashoulder width, height, arm length, and/or the like may be used, atleast in part, to determine an identity of the person 4502. The identityof the person 4502 is used to assign the beverage 4540 to the correctperson 4502, as described with respect to step 4616 below. In somecases, the identity of the person 4502 may be used to verify that thesame person 4502 is identified at different time points (e.g., at thestart of the event detected at step 4604 and at the time the cup 4508 isremoved from the second zone 4526). The beverage 4540 may only beassigned at step 4616 (see below) if the same person 4502 is determinedto have initiated the event (e.g., operated the dispensing mechanism4506) detected at step 4604 and removed the cup 4508 from the secondzone 4526 (see step 4612).

At step 4610, the beverage assignment subsystem 4530 determines, in afirst one or more images 4528 associated with a start of the detectedevent (e.g., from the first set of images 4528 determined at step 4604),that both a hand of the person 4502 enters the first zone 4524 and thecup 4508 is placed in the second zone 4526. For example, the position ofthe hand of the person 4502 may be determined from the pose determinedat step 4506. If both of (1) the hand of the person 4502 entering thefirst zone 4524 and (2) the cup 4508 being placed in the second zone4526 are not detected at step 4610, the beverage assignment subsystem4530 may return to the start of method 4600 to continue receiving images4528. If the both of (1) the hand of the person 4502 entering the firstzone 4524 and (2) the cup 4508 being placed in the second zone 4526 aredetected, the beverage assignment subsystem 4530 proceeds to step 4612.

At step 4612, the beverage assignment subsystem 4530 determines if, at asubsequent time, the cup 4508 is removed from the second zone 4526. Forexample, the beverage assignment subsystem 4530 may determine if the cup4508 is no longer detected in the second zone 4526 in images 4528corresponding to a subsequent time to the event detected at step 4604and/or if movement of the cup 4508 out of the second zone 4526 isdetected across a series of consecutive images 4528 (e.g., correspondingto the person 4502 moving the cup 4508 out of the second zone 4526). Ifthe cup 4508 is not removed from the zone 4526, the beverage assignmentsubsystem 4530 may return to the start of the method 4600 to detectsubsequent person-beverage machine interaction events. If the cup 4508is removed from the zone 4526, the beverage assignment subsystem 4530proceeds to step 4614.

At step 4614, following detecting that the cup 4508 is removed from thesecond zone 4526, the beverage assignment subsystem 4530 determineswhether the cup 4508 was in the second zone 4526 for at least athreshold length of time. For example, the beverage assignment subsystem4530 may determine a length of time during which the cup 4508 remainedin the second zone 4526. If the determined length of time is at least athreshold time, the beverage assignment subsystem 4530 may proceed tostep 4616 to assign the beverage 4540 to the person 4502 whose handentered the first zone 4524. If the cup 4508 did not remain in in thesecond zone 4526 for at least the threshold time, the beverageassignment subsystem 4530 may return to the start of the method 4600,and the beverage 4540 is not assigned to the person 4502 whose handentered the first zone 4524.

At step 4616, the beverage assignment subsystem 4530 assigns thebeverage 4540 to the person 4502 whose hand entered the first zone 4524,as determined at step 4610. For example, the beverage 4540 may beassigned to the person 4502 by adding an indicator of the beverage 4540to the digital shopping cart 4538 associated with the person 4502. Thebeverage assignment subsystem 4530 may determine properties of thebeverage 4540 and include these properties in the digital shopping cart4538. For example, the beverage assignment subsystem 4530 may determinea type of the beverage 4540. The type of the beverage 4540 may bepredetermined for the beverage machine 4504 that is viewed by theangled-view camera 4520 which views the beverage machine 4504. Forinstance, the angled-view images 4528 may be predefined as images 4528of a beverage machine 4504 known to dispense a beverage of a given type(e.g., coffee). A size of the beverage 4540 may be determined based onthe size of the cup 4508 detected in the images 4528.

In one embodiment, the beverage assignment subsystem 4530 may determinea beverage type based on the type of cup that is placed in the secondzone 4526. For example, the beverage assignment subsystem 4530 maydetermine the beverage type is coffee when a coffee cup or mug is placedin the second zone 4526. As another example, the beverage assignmentsubsystem 4530 may determine the beverage type is a frozen drink or asoft drink when a particular type of cup is placed in the second zone4526. In this example, the cup may have a particular color, size, shape,or any other detectable type of feature.

b. Detection and Assignment for “Smart” Beverage Machines

FIG. 47 illustrates a method 4700 of operating the system 4500 of FIG.45 to assign a beverage 4540 to a person 4502, based on a trigger 4512,4518 and using angled-view images 4528 captured by the angle-view sensor4520. In one embodiment, the system 4500 may be configured as acontact-less device that allows a person 4502 to order a beveragewithout contacting the beverage dispensing device. For example, thesystem 4500 may be configured to allow a person 4502 to order a drinkremotely using a user device (e.g. a smartphone or computer) and todispense the drink before the person 4502 arrives to retrieve theirbeverage. The system 4500 may use geolocation information, timeinformation, or any other suitable type of information to determine whento dispense a beverage for the person 4502. As an example, the system4500 may use geolocation information to determine when the person 4502is within a predetermined range of the beverage dispensing device. Inthis example, the system 4500 dispenses the beverage when the person4502 is within the predetermined range of the beverage dispensingdevice. As another example, the system 4500 may be time information todetermine a scheduled time for dispensing the beverage for the person4502. In this example, the person 4502 may specify a time when theyorder their beverage. The system 4500 will schedule the beverage to bedispensed by the requested time. In other examples, the system 4500 mayuse any other suitable type or combination of information to determinewhen to dispense a beverage for the person 4502.

The method 4700 may begin at step 4702 where a flow trigger 4512 and/ora device trigger 4518 are received. For example, a flow trigger 4518 maybe provided based on a measured flow of the beverage 4540 by a flowmeter 4510 out of the dispensing mechanism 4506. The flow trigger 4512may include a time when the flow of the beverage 4540 started and/orstopped, a volume of the beverage 4540 that was dispensed, and/or a typeof the beverage 4540 dispensed. A device trigger 4518 may becommunicated by a device computer 4514 of the beverage machine 4504(e.g., from an API 4516 of the device computer 4514), as described abovewith respect to FIG. 45 . The device trigger 4518 may indicate a timewhen the flow of the beverage 4540 started and/or stopped, a volume ofthe beverage 4540 that was dispensed, and/or a type of the beverage 4540dispensed. Information from one or both of the triggers 4512, 4518 maybe used to identify appropriate images 4518 (e.g., at appropriate timesor from appropriate time intervals) to evaluate in subsequent steps ofthe method 4700 (e.g., to identify images 4528 at step 4706).Information from one or both of the triggers 4512, 4518 may also oralternatively be used, at least in part, to identify the beverage 4540that is assigned to the person 4502 at step 4714.

At step 4704, an image feed comprising the angled-view images 4528 isreceived by the beverage assignment subsystem 4530. The angled-viewimages 4528 may begin to be received in response to the trigger 4512,4518. For example, the angled-view sensor 4520 may become active andbegin capturing and transmitting angled-view images 4528 following thetrigger 4512, 4518. In some embodiments, the angled-view images 4528 maybe received after the person 4502 is within a threshold distance of thebeverage machine 4502. For instance, top-view images captured by sensors108 within the space 102 (see FIG. 1 ) may be used to determine when aproximity trigger (e.g., the proximity trigger 4002 of FIG. 40 ) shouldcause the beverage assignment subsystem 4530 to begin receivingangled-view images 4528. As described above with respect to FIG. 45 ,the angled view images 4528 are from a field-of-view 4522 thatencompasses at least a portion of the beverage machine 4504, includingthe first zone 4524 associated with operating the dispensing mechanism4506 of the beverage machine 4504 and the second zone 4526 in which thecup 4508 is placed to receive the beverage 4540 from the beveragemachine 4504.

At step 4706, the beverage assignment subsystem 4530 determinesangled-view images 4528 that are associated with the start and end of abeverage-dispensing event associated with the trigger 4512, 4518. Forexample, the beverage assignment subsystem 4530 may determine a firstone or more images 4528 associated with the start of the beverage 4540being dispensed by detecting a hand of the person 4502 entering the zone4524 in which the dispensing mechanism 4506 of the beverage machine 4504is located. In some cases, the image(s) 4528 associated with the startof the beverage 4540 being dispensed by detecting that the person 4502is within a threshold distance of the beverage machine 4504, determininga hand or wrist position of the person 4502 using pose estimationalgorithm (e.g., as described above with respect to the determination ofskeletons 3302 e and 3302 f shown in FIG. 33C and skeleton 4104 shown inFIGS. 41 and 42 ), and determining that the hand or wrist positionenters zone 4524. The beverage assignment subsystem 4530 may alsodetermine a second one or more images 4528 associated with an end of thebeverage 4540 being dispensed by detecting the hand or wrist of theperson 4502 exiting the zone 4526 (e.g., based on pose estimation). Insome cases, the second image(s) 4528 may be determined by determiningthat the hand or wrist position of the person 4502 exits the zone 4524.

At step 4708, the beverage assignment subsystem 4530 determines, basedon the images 4528 identified at step 4706 associated with the start ofthe beverage being dispensed, a first identifier of the person 4502whose hand entered the zone 4524. For example, the person 4502 may beidentified based on features and/or descriptors, such as height, haircolor, clothing properties, and/or the like of the person 4502 (see,e.g., FIGS. 27-32 and 46 and corresponding descriptions above). At step4710, the beverage assignment subsystem 4530 determines, based on theimages 4528 identified at step 4706 associated with the end of thebeverage being dispensed, a second identifier of the person 4502 whosehand exited the zone 4526 (e.g., to remove the cup 4508). The person maybe identified using the same approach described above with respect tostep 4708.

At step 4712, the beverage assignment subsystem 4530 determines if thefirst identifier from step 4708 is the same as the second identifierfrom step 4710. In other words, the beverage assignment subsystem 4530determines whether the same person 4502 began dispensing the beverage4540 and removed the cup 4508 containing the beverage 4540 after thebeverage 4540 was dispensed. If the first identifier from step 4708 isthe same as the second identifier from step 4710, the beverageassignment subsystem 4530 proceeds to step 4714 and assigns the beverage4540 to the person 4502. However, if the first identifier from step 4708is not the same as the second identifier from step 4710, the beverageassignment subsystem 4530 may proceed to step 4716 to flag the event forfurther review and/or beverage assignment. For example, the beverageassignment subsystem 4530 may track movement of the person 4502 and/orcup 4508 through the space 102 after the cup 4508 is removed from zone4526 to determine if the beverage 4540 should be assigned to the person4502, as described above with respect to FIGS. 36A-B and 37.

At step 4714, the beverage 4540 is assigned to the person 4502. Forexample, the beverage 4540 may be assigned to the person 4502 by addingan indicator of the beverage 4540 to the digital shopping cart 4538associated with the person 4502. The beverage assignment subsystem 4530may determine properties of the beverage 4540 and include theseproperties in the digital shopping cart 4538. For example, theproperties may be determined based on the trigger 4512, 4518. Forinstance, a device trigger 4518 may include an indication of a drinktype and size that was dispensed, and the assigned beverage 4540 mayinclude these properties. In some cases, before assigning the beverage4540 to the person 4502, the beverage assignment subsystem 4530 maycheck that a time interval between the start of the beverage 4540 beingdispensed and the end of the beverage 4540 being dispensed is at least athreshold value (e.g., to verify that the dispensing process lasted longenough for the beverage 4540 to have been dispensed).

Sensor Mounting Assembly

FIGS. 49-56 illustrate various embodiments of a sensor mounting assemblythat is configured to support a sensor 108 and its components within aspace 102. For example, the sensor mounting assembly may be used tomount a sensor 108 near a ceiling of a store. In other examples, thesensor mounting assembly may be used to mount a sensor 108 in any othersuitable type of location. The sensor mounting assembly generallycomprises a sensor 108, a mounting ring 4804, a faceplate support 4802,and a faceplate 5102. Each of these components is described in moredetail below. FIG. 48 is a perspective view of an embodiment of afaceplate support 4802 being installed into a mounting ring 4804. Themounting ring 4804 provides an interface that allows a sensor 108 to beintegrated within a structure that can be installed near a ceiling ofspace 102. Examples of a structure include, but are not limited to,ceiling tiles, a housing (e.g. a canister), a rail system, or any othersuitable type of structure for mounting sensors 108. The mounting ring4804 comprises an opening 4806 and a plurality of threads 4808 disposedwithin the opening 4806 of the mounting ring 4804. The opening 4806 issized and shaped to allow the faceplate support 4802 to be installedwithin the opening 4806.

The faceplate support 4802 comprises a plurality of threads 4810 and anopening 4812. The threads 4810 of the faceplate support 4802 areconfigured to engage the threads 4808 of the mounting ring 4804 suchthat the faceplate support 4802 can be threaded into the mounting ring4804. Threading the faceplate support 4802 to the mounting ring 4804couples the two components together to secure the faceplate support 4802within the mounting ring 4804. The opening 4812 is sized and shaped toallow a faceplate 5102 and sensor 108 to be installed within the opening4812. Additional information about the faceplate 5102 is described belowin FIGS. 51-55 .

FIG. 49 is a perspective view of an embodiment of a mounting ring 4804.In FIG. 49 , the mounting ring 4804 is shown integrated with a ceilingtile 4902. In some embodiments, the mounting ring 4804 further comprisesa recess 4904 disposed circumferentially about the opening 4806 of themounting ring 4804. In this configuration, the recess 4904 is configuredsuch that the faceplate support 4802 can be installed substantiallyflush with the ceiling tile 4902.

FIG. 50 is a perspective view of an embodiment of a faceplate support4802. In some embodiments, the faceplate support 4802 further comprisesa lip 5002 disposed about the opening 4812 of the faceplate support4802. The lip 5002 is configured to support a faceplate 5102 by allowingthe faceplate 5102 to rest on top of the lip 5002. In thisconfiguration, the faceplate 5102 may be allowed to rotate about theopening 4812 of the faceplate support 4802. This configuration allowsthe field-of-view of the sensor 108 to be rotated by rotating thefaceplate 5102 after the sensor 108 has been installed onto thefaceplate 5102. In some embodiments, the faceplate support 4802 may beconfigured to allow the faceplate 5102 to freely rotate about theopening 4812 of the faceplate support 4802. In other embodiments, thefaceplate support 4802 may be configured to allow the faceplate 5102 torotate to fixed angles about the opening 4812 of the faceplate support4802.

In some embodiments, the faceplate support 4802 may further compriseadditional threads 5004 that are configured to allow additionalcomponents to be coupled to the faceplate support 4802. For example, ahousing or cover may be coupled to the faceplate support 4802 bythreading onto the threads 5004 of the faceplate support 4802. In otherexamples, any other suitable type of component may be coupled to thefaceplate support 4802.

FIG. 51 is a perspective view of an embodiment of a faceplate 5102. Thefaceplate 5102 is configured to support a sensor 108 and its components.The faceplate 5102 may comprise one or more interfaces or surfaces thatallow a sensor 108 and its components to be coupled or mounted to thefaceplate 5102. For example, the faceplate 5102 may comprise a mountingsurface 5106 for a sensor 108 that is configured to face an upwarddirection or a ceiling of a space 102. As another example, the faceplate5102 may comprise a mounting surface 5108 for a sensor 108 that isconfigured to face a downward direction or a ground surface of a space102. The faceplate 5102 further comprises an opening 5104. The opening5104 may be sized and shaped to support various types of sensors 108.Examples of different types of faceplate 5102 configurations aredescribed below in FIGS. 52-55 .

Examples of an Installed Sensor

FIGS. 52 and 53 combine to show different perspectives of an embodimentof a sensor 108 installed onto a faceplate 5102. FIG. 52 is a topperspective view of a sensor 108 installed onto a faceplate 5102. FIG.53 is a bottom perspective view of the sensor 108 installed onto thefaceplate 5102. In this example, the sensor 108 is coupled to faceplate5102 and oriented with a field-of-view in a downward direction throughthe opening 5104 of the faceplate 5102. The sensor 108 may be configuredto capture two-dimensional and/or three-dimensional images. For example,the sensor 108 may be a three-dimensional camera that is configured tocapture depth information. In other examples, the sensor 108 may be atwo-dimensional camera that is configured to capture RGB, infrared, orintensity images. In some examples, the sensor 108 may be configuredwith more than one camera and/or more than one type of camera. Thesensor 108 may be coupled to the faceplate 5102 using any suitable typeof brackets, mounts, and/or fasteners.

In this example, the faceplate further comprises a support 5202. Thesupport 5202 is an interface for coupling additional components. Forexample, one or more supports 5202 may be coupled to printed circuitboards 5204, microprocessors, power supplies, cables, or any othersuitable type of component that is associated with the sensor 108. Here,the support 5202 is coupled to a printed circuit board 5204 (e.g. amicroprocessor) for the sensor 108. In FIG. 52 , the printed circuitboard 5204 is shown in a vertical orientation. In other examples, theprinted circuit board 5204 may be positioned in a horizontalorientation.

FIGS. 54 and 55 combine to show different perspectives of anotherembodiment of a sensor 108 installed onto a faceplate 5102. FIG. 54 is atop perspective view of a sensor 108 installed onto a faceplate 5102.FIG. 55 is a bottom perspective view of the sensor 108 installed ontothe faceplate 5102. In this configuration, the sensor 108 is adome-shaped camera that is attached to a mounting surface of thefaceplate 5102 that faces a ground surface. In other words, the sensor108 is configured to hang from beneath the faceplate 5102. In thisexample, the faceplate 5102 comprises a support 5402 for mounting thesensor 108. In other examples, the sensor 108 may be coupled to thefaceplate 5102 using any suitable type of brackets, mounts, and/orfasteners.

Adjustable Positioning System

FIG. 56 is a perspective view of an embodiment of a sensor assembly 5602installed onto an adjustable positioning system 5600. The adjustablepositioning system 5600 comprises a plurality of rails (shown as rails5604 and 5606) that are configured to hold a sensor assembly 5602 in afixed location with respect to a global plane 104 of a space 102. Forexample, the adjustable positioning system 5600 may be installed in astore to position a plurality of sensor assemblies 5602 near a ceilingof the store. The adjustable positioning system 5600 allows the sensorassemblies 5602 to be distributed to such that these sensor assemblies5602 are able to collectively provide coverage for the store. In thisexample, each sensor assembly 5602 is integrated within a canister orcylindrical housing. In other examples, a sensor assembly 5602 may beintegrated within any other suitable shape housing. For instance, asensor assembly 5602 may be integrated within a cuboid housing, aspherical housing, or any other suitable shape housing.

In this example, rails 5604 are configured to allow a sensor assembly5602 to be repositioned along an x-axis of the global plane 104. Rails5606 are configured to allow a sensor assembly 5602 to be repositionedalong a y-axis of the global plane 104. In one embodiment, the rails5604 and 5606 may comprise a plurality of notches or recesses that areconfigured to hold a sensor assembly 5602 at a particular location. Forexample, a sensor assembly 5602 may comprise one or more pins orinterfaces that are configured to engage a notch or recess of a rail. Inthis example, the position of the sensor assembly 5602 becomes fixedwith respect to global plane 104 when the sensor assembly 5602 iscoupled to the rails 5604 and 5604 of the adjustable positioning system5600. The sensor assembly 5602 may be repositioned at a later time bydecoupling the sensor assembly 5602 from the rails 5604 and 5604,repositioning the sensor assembly 5602, and recoupling the sensorassembly 5602 to the rails 5604 and 5604. In other examples, theadjustable positioning system 5600 may use any other suitable type ofmechanism for coupling a sensor assembly 5602 to the rails 5604 and5606.

Position Sensors

In one embodiment, a position sensor 5610 may be coupled to each of thesensor assemblies 5602. Each position sensor 5610 is configured tooutput a location for a sensor 108. For example, a position sensor 5610may be an electronic device configured to output an (x,y) coordinate fora sensor 108 that described the physical location of the sensor 108 withrespect to the global plane 104 and the space 102. Examples of aposition sensor include, but are not limited to, Bluetooth beacons or anelectrical contact-based circuit. In some examples, a position sensor5610 may be further configured to output a rotation angle for a sensor108. For instance, the position sensor 5610 may be configured todetermine a rotation angle of a sensor 108 with respect to the groundplane 104 and to output the determined rotation angle. The positionsensor 5610 may use an accelerometer, a gyroscope, or any other suitabletype of mechanism for determining a rotation angle for a sensor 108. Inother examples, the position sensor 5610 may be replaced with marker ormechanical indicator that indicates the location and/or the rotation ofa sensor 108.

Draw Wire Encoder System

FIGS. 57 and 58 illustrate an embodiment of a draw wire encoder system5700 that can be employed for generating a homography 118 for a sensor108 of the tracking system 100. The draw wire encoder system 5700 may beconfigured to provide an autonomous process for repositioning one ormore markers 5708 within a space 102 and capturing frames 302 of themarker 5708 that can be used to generate a homography 118 for a sensor108. An example of a process for using the draw wire encoder system 5700is described in FIG. 59 .

Draw Wire Encoder System Overview

FIG. 57 is an overhead view of an example of a draw wire encoder system5700. The draw wire encoder system 5700 comprises a plurality of drawwire encoders 5702. Each of the draw wire encoders 5702 is a distancemeasuring device that is configured to measure the distance between adraw wire encoder 5702 and an object that is operably coupled to thedraw wire encoder 5702. The locations of the draw wire encoders 5702 isknown and fixed within the global plane 104. In one embodiment, a drawwire encoder 5702 comprises a housing, a retractable wire 5706 that isstored within the housing, and an encoder. The encoder is configured tooutput a signal or value that corresponds with an amount of theretractable wire 5706 that extends outside of the housing. In otherwords, the encoder is configured to report a distance between a drawwire encoded 5702 and an object (e.g. platform 5704) that is attached tothe end of the retractable wire 5706.

In one embodiment, the draw wire encoders 5702 are configured towirelessly communicate with sensors 108 and the tracking system 100. Forexample, the draw wire encoders 5702 may be configured to receive datarequests from a sensor 108 and/or the tracking system 100. In responseto receiving a data request, a draw wire encoder 5702 may be configuredto send information about the distance between the draw wire encoder5702 and a platform 5704. The draw wire encoders 5702 may communicatewirelessly using Bluetooth, WiFi, Zigbee, Z-wave, and any other suitabletype of wireless communication protocol.

The draw wire encoder system 5700 further comprises a moveable platform5704. An example of the platform 5704 is described in FIG. 58 . Theplatform 5704 is configured to be repositionable within a space 102. Forexample, the platform 5704 may be physically moved to differentlocations within a space 102. As another example, the platform 5704 maybe remotely controlled to reposition the platform 5704 within a space102. For instance, the platform 5704 may be integrated with aremote-controlled device that can be repositioned within a space 102 byan operator. As another example, the platform 5704 may an autonomousdevice that is configured to reposition itself within a space 102. Inthis example, the platform 5704 may be configured to freely roam a space102 or may be configured to follow a predetermined path within a space102. In FIG. 57 , the draw wire encoder system 5700 is configured with asingle platform 5704. In other examples, the draw wire encoder system5700 may be configured to use a plurality of platforms 5704.

The platform 5704 is coupled to the retractable wires 5706 of the drawwire encoders 5702. The draw wire encoder system 5700 is configured todetermine the location of the platform 5704 within the global plane 104of the space 102 based on the information that is provided by the drawwire encoders 5702. For example, the draw wire encoder system 5700 maybe configured to use triangulation to determine the location of theplatform 5704 within the global plane 104. For instance, the draw wireencoder system 5700 may determine the location of the platform 5704within the global plane 104 using the following expressions:

$\gamma = {\cos^{- 1}\left( \frac{{D\; 1^{2}} + {D\; 3^{2}} - {D\; 2^{2}}}{2*D\; 1*D\; 3} \right)}$y = D 1 * cos (90 − γ) x = D 1 * cos (90 − γ)where D1 is the distance between a first draw wire encoder 5702A and theplatform 5704, D2 is the distance between a second draw wire encoder5702B and the platform 5704, D3 is the distance between the first drawwire encoder 5702A and the second draw wire encoder 5702B, x is thex-coordinate of the platform 5704 in the global plane 104, and y is they-coordinate of the platform 5704 in the global plane 104. In otherexamples, the draw wire encoder system 5700 may compute the location ofthe platform 5704 within the global plane 104 using any other suitabletechnique. In other examples, the draw wire encoder system 5700 may beconfigured to use any other suitable mapping function to determine thelocation of the platform 5704 within the global plane 104 based on thedistances reported by the draw wire encoders 5702.

The platform 5704 comprises one or more markers 5708 that are visible toa sensor 108. Examples of markers 5708 include, but are not limited, totext, symbols, encoded images, light sources, or any other suitable typeof marker that can be detected by a sensor 108. Referring to the examplein FIGS. 57 and 58 , the platform 5704 comprises an encoded imagedisposed on a portion of the platform 5704 that is visible to a sensor108. As another example, the platform 5704 may comprise a plurality ofencoded images that are disposed on a portion of the platform 5704 thatis visible to a sensor 108. As another example, the platform 5704 maycomprise a light source (e.g. an infrared light source) that is disposedon a portion of the platform 5704 that is visible to a sensor 108. Inother examples, the platform 5704 may comprise any other suitable typeof marker 5708.

As an example, a sensor 108 may capture a frame 302 that includes amarker 5708. The tracking system 100 will process the frame 302 todetect the marker 5708 and to determine a pixel location 5710 within theframe 302 where the marker 5708 is located. The tracking system 100 usespixel locations 5710 of a plurality of markers 5708 with thecorresponding physical locations of the markers 5708 within the globalplane 104 to generate a homography 118 for the sensor 108. An example ofthis process is described in FIG. 59 .

In other examples, the tracking system 100 may use any other suitabletype of distance measuring device in place of the draw wire encoders5702. For example, the tracking system 100 may use Bluetooth beacons,Global Position System (GPS) sensors, Radio-Frequency Identification(RFID) tags, millimeter wave (mmWave) radar or laser, or any othersuitable type of device for measuring distance or providing locationinformation. In these examples, each distance measuring device isconfigured to measure and output a distance between the distancemeasuring device and an object that is operably coupled to the distancemeasuring device.

FIG. 58 is a perspective view of a platform 5704 for a draw wire encodersystem 5700. In this example, the platform 5704 comprises a plurality ofwheels 5802 (e.g. casters). In other examples, the platform 5704 maycomprise any other suitable type of mechanism that allows the platform5704 to be repositioned within a space 102. In some embodiments, theplatform 5704 be configured with an adjustable height 5804 that allowsthe platform 5704 to change the elevation of the markers 5708. In thisconfiguration, the height 5804 of the marker 5708 can be adjusted whichallows the tracking system 100 to generate homographies 118 at more thanone elevation level for a space being mapped by a sensor 108. Forexample, the tracking system 100 may generate homographies 118 for morethan one plane (e.g. x-y plane) or cross-section along the vertical axis(e.g. the z-axis) dimension. This process allows the tracking system 100to use more than one elevation level when determining the physicallocation of an object or person within the space 102, which improves theaccuracy of the tracking system 100.

Sensor Mapping Process Using a Draw Wire Encoder System

FIG. 59 is a flowchart of an embodiment of a sensor mapping method 5900using a draw wire encoder system 5700. The tracking system 100 mayemploy method 5900 to autonomously generate a homography 118 for asensor 108. This process involves using the draw wire encoder system5700 to reposition markers 304 within the field of view of the sensor108. Using the draw wire encoder system 5700, the tracking system 100 isable to simultaneously obtain location information for the markers 304which reduces the amount of time it takes the tracking system 100 togenerate a homography 118. Using the draw wire encoder system 5700, thetracking system 100 may also autonomously reposition the markers 304within the field of view of the sensor 108 improves the efficiency ofthe tracking system 100 and further reduces the amount of time it takesto generate a homography 118. The following is a non-limiting of aprocess for generating a homography 118 for a single sensor 108. Thisprocess can be repeated for generating a homography 118 for othersensors 108. In this example, the tracking system 100 employs draw wireencoders 5702. However, in other examples, the tracking system 100 mayemploy a similar process using any other suitable type of distancemeasuring device in place of the draw wire encoders 5702.

At step 5902, the tracking system 100 receives a frame 302 with a marker5708 at a location within the space 102 from a sensor 108. Here, thetracking system 100 receives a frame 302 from a sensor 108. For example,the sensor 108 may capture an image or frame 302 of the global plane 104for at least a portion of the space 102. The frame 302 may comprise oneor more markers 5708.

At step 5904, the tracking system 100 determine pixel locations 5710 inthe frame 302 for the markers 5708. In one embodiment, the trackingsystem 100 uses object detection to identify markers 5708 within theframe 302. For example, the markers 5708 may have known features (e.g.shape, pattern, color, text, etc.) that the tracking system 100 cansearch for within the frame 302 to identify a marker 5708. Referring tothe example in FIG. 57 , the marker 5708 comprises an encoded image. Inthis example, the tracking system 100 may search the frame 302 forencoded images that are present within the frame 302. The trackingsystem 100 will identify the marker 5708 and any other markers 5708within the frame 302. In other examples, the tracking system 100 mayemploy any other suitable type of image processing technique foridentifying markers 5708 within the frame 302. After identifying amarker 5708 within the frame 302, the tracking system 100 will determinea pixel location 5701 for the marker 5708. The pixel location 5710comprises a pixel row and a pixel column indicating where the marker5708 is located in the frame 302. The tracking system 100 may repeatthis process for any suitable number of markers 304 that are within theframe 302.

At step 5906, the tracking system 100 determines (x,y) coordinates 304of the markers 5708 within the space 102. In one embodiment, thetracking system 100 may send a data request to the draw wire encoders5702 to request location information for the platform 5704. The trackingsystem 100 may then determine the location of a marker 5708 based on thelocation of the platform 5704. For example, the tracking system 100 mayuse a known offset between the location of the markers 5708 and locationwhere the platform 5704 is connected to the draw wire encoders 5702. Inthe example shown in FIG. 57 , the marker 5708 is positioned with zerooffset from the location where the draw wire encoders 5702 are connectedto the platform 5704. In other words, the marker 5708 is positioneddirectly above where the platform 5704 is connected to the draw wireencoders 5702. In this example, the tracking system 100 may send a firstdata request to the first draw wire encoder 5702A to request locationinformation for the marker 5708. The tracking system 100 may also send asecond data request to the second draw wire encoder 5702B to requestlocation information for the marker 5708. The tracking system 100 maysend the data requests to the draw wire encoders 5702 using any suitabletype of wireless or wired communication protocol.

The tracking system 100 may receive a first distance from the first drawwire encoder 5702A in response to sending the first data request. Thefirst distance corresponds to the distance between the first draw wireencoder 5702A and the platform 5704. In this example, since the marker5708 has a zero offset with where the platform 5704 is tethered to thefirst draw wire encoder 5704A, the first distance also corresponds withthe location of the marker 5708. In other examples, the tracking system100 may determine the location of a marker 5708 based on an offsetdistance between the location of the marker 5708 and the location wherethe platform 5704 is tethered to a draw wire encoder 5702. The trackingsystem 100 may receive a second distance from the second draw wireencoder 5702B in response to the second data request. The trackingsystem 100 may use a similar process to determine the location of themarker 5708 based on the second distance. The tracking system 100 maythen use a mapping function to determine an (x,y) coordinate for themarker 5708 based on the first distance and the second distance. Forexample, the tracking system 100 may use the mapping function describedin FIG. 57 . In other examples, the tracking system 100 may use anyother suitable mapping function to determine an (x,y) coordinate for themarker 5708 based on the first distance and the second distance.

In some embodiments, the draw wire encoders 5702 may be configured todirectly output an (x,y) coordinate for the marker 5708 in response to adata request from the tracking system 100. For example, the trackingsystem 100 may send data requests to the draw wire encoders 5702 and mayreceive location information for the platform 5704 and/or the markers5708 in response to the data requests.

At step 5908, the tracking system 100 determines whether to captureadditional frames 302 of the markers 5708. Each time a marker 5708 isidentified and located within the global plane 104, the tracking system100 generates an instance of marker location information. In oneembodiment, the tracking system 100 may count the number of instances ofmarker location information that have been recorded for markers 5708within the global plane 104. The tracking system 100 then determineswhether the number of instances of marker location information isgreater than or equal to a predetermined threshold value. The trackingsystem 100 may compare the number of instances of marker locationinformation to the predetermined threshold value using a process similarto the process described in step 210 of FIG. 2 . The tracking system 100determines to capture additional frames 5708 in response to determiningthat the number of instances of marker location information is less thanthe predetermined threshold value. Otherwise, the tracking system 100determines not to capture additional frames 302 in response todetermining that the number of instances of marker location informationis greater than or equal to the predetermined threshold value.

The tracking system 100 returns to step 5902 in response to determiningto capture additional frames 302 of the marker 5708. In this case, thetracking system 100 will collect additional frames 302 of the marker5708 after the marker 5708 is repositioned within the global plane 104.This process allows the tracking system 100 to generate additionalinstances of marker location information that can be used for generatinga homography 118. The tracking system 100 proceeds to step 5910 inresponse to determining not to capture additional frames 302 of themarker 5708. In this case, the tracking system 100 determines that asuitable number of instances of marker location information have beenrecorded for generating a homography 118.

At step 5910, the tracking system 100 generates a homography 118 for thesensor 108. The tracking system 100 may generate a homography 118 usingany of the previously described techniques. For example, the trackingsystem 100 may generate a homography 118 using the process described inFIGS. 2 and 6 . After generating the homography 118, the tracking system100 may store an association between the sensor 108 and the generatedhomography 118. This process allows the tracking system 100 to use thegenerated homography 118 for determining the location of objects withinthe global plane 104 using frames 302 from the sensor 108.

Food Detection Process

FIG. 60 is a flowchart of an object tracking method 6000 for thetracking system 100. In one embodiment, the tracking system 100 mayemploy method 6000 for detecting when a person removes or replaces item6104 from a food rack 6102. In a first phase of method 6000, thetracking system 100 uses a region-of-interest (RIO) marker to define aROI or zone 6108 within the field of view of a sensor 108. This phaseallows the tracking system 100 to reduce the search space when detectingand identifying objects within the field of view of a sensor 108. Thisprocess improves the performance of the system by reducing search timeand reducing the utilization of processing resources. In a second phaseof method 6000, the tracking system 100 uses the previously define zone6108 to detect and identify objects that a person removes or replacesfrom a food rack 6102.

Defining a Region-of-Interest

At step 6002, the tracking system 100 receives a frame 302 from a sensor108. Referring to FIG. 61 as an example, this figure illustrates anoverheard view of a portion of space 102 (e.g. a store) with a food rack6102 that is configured to store a plurality of items 6104 (e.g. food orbeverage items). For example, the food rack 6102 may be configured tostore hot food, fresh food, canned food, frozen food, or any othersuitable types of items 6104. In this example, the sensor 108 may beconfigured to capture frames 302 of a top view or a perspective view ofa portion of the space 102. The sensor 108 is configured to captureframes 302 adjacent to the food rack 6102.

Returning to FIG. 60 at step 6004, the tracking system 100 detects anROI marker 6106 within the frame 302. The ROI marker 6106 is a markerthat is visible to the sensor 108. Examples of a ROI marker 6106include, but are not limited to, a colored surface, a patterned surface,an image, an encoded image, or any other suitable type of marker that isvisible to the sensor 108. Returning to the example in FIG. 61 , the ROImarker 6106 may be a removable surface (e.g. a mat) that comprises apredetermined color or pattern that is visible to the sensor 108. Inthis example, the ROI marker 6106 is positioned near an access point6110 (e.g. a door or opening) where a customer would reach to access theitems 6104 in the food rack 6102. The tracking system 100 may employ anysuitable type of image processing technique to identify the ROI marker6106 in the frame 302. For example, the ROI marker 6106 may be apredetermined color. In this example, the tracking system 100 may applypixel value thresholds to identify the predetermined color and the ROImarker 6106 within the frame 302. As another example, the ROI marker6106 may comprise a predetermined pattern or image. In this example, thetracking system 100 may use pattern or image recognition to identify theROI marker 6106 within the frame 302.

In some embodiments, the tracking system 100 may apply an affinetransformation to the detected ROI marker 6106 to correct forperspective view distortion. Referring to FIG. 62 , an example of askewed ROI marker 6106A is illustrated. In this example, the ROI marker6106A may be skewed because the sensor 108 is configured to capture aperspective or angled view of the space 102 with the ROI marker 6106A.In this case, the tracking system 100 may apply an affine transformationmatrix to frame 302 that includes the ROI marker 6106A to correct theskewing of the ROI marker 6106A. In other words, the tracking system 100may apply an affine transformation to the ROI marker 6106A to change theshape of the ROI marker 6106A into a rectangular shape. An example of acorrected ROI marker 6106B is illustrated in FIG. 62 . In oneembodiment, an affine transformation matrix comprises a combinationtranslation, rotation, and scaling coefficients that reshape the RIOmarker 6106A within the frame 302. The tracking system 100 may employany suitable technique for determining affine transformation matrixcoefficients and applying an affine transformation matrix to the frame302. This process assists the tracking system 100 later when determiningthe pixel locations of the ROI marker 6106 with the frame 302.

Returning to FIG. 60 at step 6006, the tracking system 100 identifiespixel locations in the frame 302 corresponding with the ROI marker 6106.Here, the tracking system 100 identifies the pixels within the frame 302that correspond with the ROI marker 6106. The tracking system 100 thendetermines the pixel locations (i.e. the pixel columns and pixel rows)that correspond with the identified pixels within the frame 302.

At step 6008, the tracking system 100 defines a zone 6108 at the pixellocations for the sensor 108. The zone 6108 corresponds with a range ofpixel columns and pixels rows that correspond with the pixel locationswhere the ROI marker 6106 was detected. Here, the tracking system 100defines these pixel locations as a zone 6108 for subsequent frames 302from the sensor 108. This process defines a subset of pixel locationswithin frames 302 from the sensor 108 where an object is most likely tobe detected. By defining a subset of pixel locations within frames 302from the sensor, this process allows the tracking system 100 to reducethe search space when detecting and identifying objects near the foodrack 6102.

Object Tracking Using the Region-of-Interest

After the tracking system 100 defines the zone 6110 within the frame 302from the sensor 108, the ROI marker 6106 may be removed from the space102 and the tracking system 100 will begin detecting and identifyingitems 6104 that are removed from the food rack 6102. At step 6010, thetracking system 100 receives a new frame 302 from the sensor 108. In oneembodiment, the tracking system 100 is configured to periodicallyreceive frames 302 from the sensor 108. For example, the sensor 108 maybe configured to continuously capture frames 302. In other embodiments,the tracking system 100 may be configured to receive a frame 302 fromthe sensor 108 in response to a triggering event. For example, the foodrack 6102 may comprise a door sensor configured to output an electricalsignal when a door of the food rack 6102 is opened. In this example, thetracking system 100 may detect that the door sensor was triggered to anopen position and receives the new frame 302 from the sensor 108 inresponse to detecting the triggering event. As another example, thetracking system 100 may be configured to detect motion based ondifferences between subsequent frames 302 from the sensor 108 a striggering event. As another example, the tracking system 100 may beconfigured to detect vibrations at or near food rack 6102 using anaccelerometer. In other examples, the tracking system 100 may beconfigured to use any other suitable type of triggering event.

At step 6012, the tracking system 100 detects an object within the zone6108 of the frame 302. The tracking system 100 may monitor the pixellocations within the zone 6108 to determine whether an item 6104 ispresent within the zone 6108. In one embodiment, the tracking system 100detect an item 6104 within the zone 6108 by detecting motion within thezone 6108. For example, the tracking system 100 may compare subsequentframes 302 from the sensor 108. In this example, the tracking system 100may detect motion based on differences between subsequent frames 302.For example, the tracking system 100 may first receive a frame 302 thatdoes not include an item 6104 within the zone 6108. In a subsequentframe 302, the tracking system 100 may detect that the item 6104 ispresent within the zone 6108 of the frame 302. In other examples, thetracking system 100 may detect the item 6104 within the zone 6108 usingany other suitable technique.

At step 6014, the tracking system 100 identifies the object within thezone 6108 of the frame 302. In one embodiment, the tracking system 100may use image processing to identify the item 6104. For example, thetracking system 100 may search within the zone 6108 of the frame 302 forknown features (e.g. shapes, patterns, colors, text, etc.) thatcorrespond with a particular item 6104.

In another embodiment, the sensor 108 may be configured to capturethermal or infrared frames 302. In this example, each pixel value in aninfrared frame 302 may correspond with a temperature value. In thisexample, the tracking system 100 may identify a temperature differentialwithin the infrared frame 302. The temperature differential within theinfrared frame 302 can be used to locate an item 6104 within the frame302 since the item 6104 will typically be cooler or hotter than ambienttemperatures. The tracking system 100 may identify the pixel locationswithin the frame 302 that are greater than a temperature threshold whichcorresponds with the item 6104.

After identifying pixel locations within the frame 302, the trackingsystem 100 may generate a binary mask 6402 based on the identified pixellocation. The binary mask 6402 may have the same pixel dimensions as theframe 302 or the zone 6108. In this example, the binary mask 6402 isconfigured such that pixel locations outside of the identified pixellocations corresponding with the item 6104 are set to a null value. Thisprocess generates a sub-ROI by removing the information from pixellocations outside of the identified pixel locations corresponding withthe item 6104 which isolates the pixel locations associated with theitem 6104 in the frame 302. After generating the binary mask 6402, thetracking system 100 applies the binary mask 6402 to the frame 302 toisolates the pixel locations associated with the item 6104 in the frame302. An example of applying the binary mask 6402 to the frame 302 isshown in FIG. 64 . In this example, the tracking system 100 firstremoves pixel locations outside of the zone 6108 to reduce the searchspace for locating the item 6104 within the frame 302. The trackingsystem 100 then applies the binary mask 6402 to the remaining pixellocations to isolate the item 6104 within the frame 302. After isolatingthe item 6104 within the frame 302, the tracking system 100 may then useany suitable object detection technique to identify the item 6104. Inother embodiments, the tracking system 100 may use any other suitabletechnique to identify the item 6104.

Returning to FIG. 60 at step 6016, the tracking system 100 identifies aperson 6302 within the frame 302. In one embodiment, the tracking system100 identifies the person 6302 that is closest to the food rack 6102 andthe zone 6108 of the frame 302. For example, the tracking system 100 maydetermine a pixel location in the frame 302 for the person 6302. Thetracking system 100 may determine a pixel location for the person 6302using a process similar to the process described in step 1004 of FIG. 10. The tracking system 100 may use a homography 118 that is associatedwith the sensor 108 to determine an (x,y) coordinate in the global plane104 for the person 6302. The homography 118 is configured to translatebetween pixel locations in the frame 302 and (x,y) coordinates in theglobal plane 104. The homography 118 is configured similar to thehomography 118 described in FIGS. 2-5B. As an example, the trackingsystem 100 may identify the homography 118 that is associated with thesensor 108 and may use matrix multiplication between the homography 118and the pixel location of the person 6302 to determine an (x,y)coordinate in the global plane 104. The tracking system 100 may thenidentify which person 6302 is closest to the food rack 6102 and the zone6108 based on the person's 6302 (x,y) coordinate in the global plane104.

At step 6018, the tracking system 100 determines whether the person 6302is removing the object from the food rack 6102. In one embodiment, thezone 6108 comprises an edge 6112. The edge 6112 comprises a plurality ofpixels within the zone 6112 that can be used to determine an objecttravel direction 6114. For example, the tracking system 100 may usesubsequent frames 302 from the sensor 108 to determine whether an item6104 is entering or exiting the zone 6108 when it crosses the edge 6112.As an example, the tracking system 100 may first detect an item 6104within the zone 6108 in a first frame 302. The tracking system 100 maythen determine that the item 6108 is no longer in the zone 6108 after itcrosses the edge 6112 of the zone 6108. In this example, the trackingsystem 100 determines that the item 6104 was removed from the food rack6102 based on its travel direction 6114. As another example, thetracking system 100 may first detect that an item 6104 is not presentwithin the zone 6108. The tracking system 100 may then detect that theitem 6104 is present in the zone 6108 of the frame 302 after it crossesthe edge 6112 of the zone 6108. In this example, the tracking system 100determines that the item 6104 is being returned to the food rack 6102based on its travel direction 6114.

In another embodiment, the tracking system 100 may use weight sensors110 to determine whether the item 6104 is being removed from the foodrack 6102 or the item 6104 is being returned to the food rack 6102. Asan example, the tracking system 100 may detect a weight decrease from aweight sensor 110 before the item 6104 is detected within the zone 6108.The weight decrease corresponds with the weight of the item 6104 belifted off of the weight sensor 110. In this example, the trackingsystem 100 determines that the item 6104 is being removed from the foodrack 6102. As another example, the tracking system 100 may detect aweight increase on a weight sensor 110 after the item 6104 is detectedwithin the zone 6108. The weight increase corresponds with the weight ofthe item 614 be placed onto the weight sensor 110. In this example, thetracking system 100 determines that the item 6104 is being returned tothe food rack 6102. In other embodiments, the tracking system 100 mayuse any other suitable technique for determining whether the item 6104is being removed from the food rack 6102 or the item 6104 is beingreturned to the food rack 6102.

The tracking system 100 proceeds to step 6020 in response to determiningthat the person 6302 is not removing the object. In this case, thetracking system 100 determines that the person is 6302 is putting theitem 6104 back into the food rack 6102. This means that the item 6104will need to be removed from their digital cart 1410. At step 6020, thetracking system 100 removes the object from the digital cart 1410 thatis associated with the person 6302. In one embodiment, the trackingsystem 100 may determine a number of items 6104 that are being returnedto the food rack 6102 using image processing. For example, the trackingsystem 100 may use object detection to determine a number of items 6104that are present in the frame 302 when the person 6302 returns the item6104 to the food rack 6102. In another embodiment, the tracking system100 may use weight sensors 110 to determine a number of items 6104 thatwere returned to the food rack 6102. For example, the tracking system100 may determine a weight increase amount on a weight sensor 110 afterthe person 6302 returns one or more items 6104 to the food rack 6102.The tracking system 100 may then determine an item quantity based on theweight increase amount. For example, the tracking system 100 maydetermine an individual item weight for the items 6104 that areassociated with the weight sensor 110. For instance, the weight sensor110 may be associated with an item 6104 that has an individual weight ofeight ounces. When the weight sensor 110 detects a weight increase ofsixteen ounces, the weight sensor 110 may determine that two of theitems 6104 were returned to the food rack 6102. In other embodiments,the tracking system 100 may determine a number of items 6104 that werereturned to the food rack 6102 using any other suitable type oftechnique. The tracking system 100 then removes the identified quantityof the item 6104 from the digital cart 1410 that is associated with theperson 6302.

Returning to step 6018, the tracking system 100 proceeds to step 6022 inresponse to determining that the person 6302 is removing the object. Inthis case, the tracking system 100 determines that the person 6302 isremoving the item 6104 from the food rack 6102 to purchase the item6104. At step 6022, the tracking system 100 adds the object to thedigital cart 1410 that is associated with the person 6302. In oneembodiment, the tracking system 100 may determine a number of items 6104that were removed from the food rack 6102 using image processing. Forexample, the tracking system 100 may use object detection to determine anumber of items 6104 that are present in the frame 302 when the person6302 removes the item 6104 from the food rack 6102. In anotherembodiment, the tracking system 100 may use weight sensors 110 todetermine a number of items 6104 that were removed from the food rack6102. For example, the tracking system 100 may determine a weightdecrease amount on a weight sensor 110 after the person 6302 removes oneor more items 6104 from the weight sensor 110. The tracking system 100may then determine an item quantity based on the weight decrease amount.For example, the tracking system 100 may determine an individual itemweight for the items 6104 that are associated with the weight sensor110. For instance, the weight sensor 110 may be associated with an item6104 that has an individual weight of eight ounces. When the weightsensor 110 detects a weight decrease of sixteen ounces, the weightsensor 110 may determine that two of the items 6104 were removed fromthe weight sensor 110. In other embodiments, the tracking system 100 maydetermine a number of items 6104 that were removed from the food rack6102 using any other suitable type of technique. The tracking system 100then adds the identified quantity of the item 6104 from the digital cart1410 that is associated with the person 6302.

At step 6024, the tracking system 100 determines whether to continuemonitoring. In one embodiment, the tracking system 100 may be configuredto continuously monitor for items 6104 for a predetermined amount oftime after a triggering event is detected. For example, the trackingsystem 100 may be configured to use a timer to determine an amount oftime that has elapsed after a triggering event. If an item 6104 or acustomer is not detected after a predetermined amount of time haselapsed, then the tracking system 100 may determine to discontinuemonitoring for items 6104 until the next triggering event is detected.If an item 6104 or customer is detected within the predetermined timeinterval, then the tracking system 100 may determine to continuemonitoring for items 6104.

The tracking system 100 returns to step 6010 in response to determiningto continue monitoring. In this case, the tracking system 100 returns tostep 6010 to continue monitoring frames 302 from the sensor 108 todetect when a person removes or replaces an item 6104 from the food rack6102. Otherwise, the tracking system 100 terminates method 6000 inresponse to determining to discontinue monitoring. In this case, thetracking system 100 has finished detecting and identifying items 6104and may terminate method 6000.

Sensor Reconfiguration Process

FIG. 65 is a flowchart of an embodiment of a sensor reconfigurationmethod 6500 for the tracking system 100. The tracking system 100 mayemploy method 6500 to update a homography 118 for a sensor 108 withouthaving to use markers 304 to generate a new homography 118. This processgenerally involves determining whether a sensor 108 has moved or rotatedwith respect to the global plane 104. In response to determining thatthe sensor 108 has been moved or rotated, the tracking system 100determines translation coefficients 6704 and rotation coefficients 6706based on the new orientation of the sensor 108 and uses the translationcoefficients 6704 and the rotation coefficients 6706 to update anexisting homography 118 that is associated with the sensor 108. Thisprocess improves the performance of the tracking system 100 by bypassingthe sensor calibration steps that involve placing and detecting markers304. During these sensor calibration steps, the sensor 108 is typicallytaken offline until its new homography 118 has been generated. Incontrast, this process bypasses these sensor calibration steps whichreduces the downtime of the tracking system 100 since the sensor 108does not need to be taken offline to update its homography 118.

Creating an Initial Homography

At step 6502, the tracking system 100 receives an (x,y) coordinate 6602within a space 102 for a sensor 108 at an initial location 6604.Referring to FIG. 66 as an example, the sensor 108 may be positionedwithin a space 102 (e.g. a store) with an overhead view of at least aportion of the space 102. In this example, the sensor 108 is configuredto capture frames 302 of the global plane 104 for at least a portion ofthe store. In one embodiment, the tracking system 100 may employposition sensors 5610 that are configured to output the location of thesensor 108. The position sensor 5610 may be configured similarly to theposition sensors 5610 described in FIG. 56 . In other embodiments, thetracking system 100 may be configured to receive location information(e.g. an (x,y) coordinate) for the sensor 108 from a technician or usingany other suitable technique.

Returning to FIG. 65 at step 6504, the tracking system 100 generates ahomography 118 for the sensor 108 at the initial location 6604. Thetracking system 100 may generate a homography 118 using any of thepreviously described techniques. For example, the tracking system 100may generate a homography 118 using the process described in FIGS. 2 and6 . The generated homography 118 is specific to the location and theorientation of the sensor 108 within the global plane 104.

At step 6506, the tracking system 100 associates the homography 118 withthe sensor 108 and the initial location 6604. For example, the trackingsystem 100 may store an association between the sensor 108, thegenerated homography 118, and the initial location 6604 of the sensor104 within the global plane 104. The tracking system 100 may alsoassociate the generated homography 1180 with a rotation angle for thesensor 108 or any other suitable type of information about theconfiguration of the sensor 108.

Updating the Existing Homography

At step 6508, the tracking system 100 determines whether the sensor 108has moved. Here, the tracking system 100 determines whether the positionof the sensor 108 has changed with respect to the global plane 104 bydetermining whether the sensor 108 has moved in the x-, y-, and/orz-direction within the global plane 104. For example, the sensor 108 mayhave been intentionally or unintentionally moved to a new locationwithin the global plane 104. In one embodiment, the tracking system 100may be configured to periodically sample location information (e.g. an(x,y) coordinate) for the sensor 108 from a position sensor 5610. Inthis configuration, the tracking system 100 may compare the current(x,y) coordinate for the sensor 108 to the previous (x,y) coordinate forthe sensor 108 to determine whether the sensor 108 has moved within theglobal plane 104. The tracking system 100 determines that the sensor 108moved when the current (x,y) coordinate for the sensor 108 does notmatch the previous (x,y) coordinate for the sensor 108. In otherexamples, the tracking system 100 may determine that the sensor 108 hasmoved based on an input provided by a technician or using any othersuitable technique. The tracking system 100 proceeds to step 6510 inresponse to determining that the sensor 108 has moved. In this case, thetracking system 100 will determine the new location of the sensor 108 toupdate the homography 118 that is associated with the sensor 108. Thisprocess allows the tracking system 100 to update the homography 118 thatis associated with the sensor 108 without having to use markers torecompute the homography 118.

At step 6510, the tracking system 100 receives a new (x,y) coordinate6606 within the space 102 for the sensor 108 at a new location 6608. Thetracking system 100 may determine the new (x,y) coordinate 6606 for thesensor 108 using a process similar to the process that is described instep 6502.

At step 6512, the tracking system 100 determines translationcoefficients 6704 for the sensor 108 based on a difference between theinitial location of the sensor 108 and the new location of the sensor108. The translation coefficients 6704 identify an offset between theinitial location of the sensor 108 and the new location of the sensor108. The translation coefficients 6704 may comprise an x-axis offsetvalue, a y-axis offset value, and a z-axis offset value. Returning tothe example in FIG. 66 , the tracking system 100 compares the new (x,y)coordinate 6606 of the sensor 108 to the initial (x,y) coordinate 6602to determine an offset with respect to each axis of the global plane104. In this example, the tracking system 100 determines an x-axisoffset value that corresponds with an offset 6610 with respect to thex-axis of the global plane 104. The tracking system 100 also determinesa y-axis offset value that corresponds with an offset 6612 with respectto the y-axis of the global plane 104. In other examples, the trackingsystem may also determine a z-axis offset value that corresponds with anoffset with respect to the z-axis of the global plane 104.

Returning to FIG. 65 at step 6514, the tracking system 100 updates thehomography 118 for the sensor by applying the translation coefficients6704 to the homography 118. The tracking system 100 may use thetranslation coefficients 6704 in a transformation matrix 6702 that canbe applied to the homography 118. FIG. 67 illustrates an example ofapplying a transformation matrix 6702 to a homography matrix 118 toupdate the homography matrix 118. In this example, Tx corresponds withan x-axis offset value, Ty corresponds with a y-axis offset value, andTz corresponds with a z-axis offset value. The tracking system 100includes the translation coefficients 6704 within the transformationmatrix 6702 and then uses matrix multiplication to apply the translationcoefficients 6704 and update the homography 118. The tracking system 100may set the rotation coefficients 6706 to a value of one when norotation is being applied to the homography 118.

Returning to FIG. 65 at step 6508, the tracking system 100 proceeds tostep 6516 in response to determining that the sensor 108 has not moved.In this case, the tracking system 100 determines that the sensor 108 hasnot moved when the current (x,y) coordinate for the sensor 108 matchesthe previous (x,y) coordinate for the sensor 108. The tracking system100 may then determine whether the sensor 108 has been rotated withrespect to the global plane 104. At step 6516, the tracking system 100determines whether the sensor 108 has been rotated. Returning to theexample in FIG. 66 , the sensor 108 has been rotated after the sensor108 was moved to the new (x,y) coordinate 6606. In this example, thesensor 108 was rotated about ninety degrees. In other examples, thesensor 108 may be rotated by any other suitable amount. In oneembodiment, the tracking system 100 may periodically receive locationinformation (e.g. a rotation angle) for the sensor 108 from a positionsensor 5610. For example, the position sensor 5610 may comprise anaccelerometer or gyroscope that is configured to output a rotation anglefor the sensor 108. In other examples, the tracking system 100 mayreceive a rotation angle for the sensor 108 from a technician or usingany other suitable technique. The tracking system 100 determines thatthe sensor 108 has been rotated in response to receive a rotation anglegreater than zero degrees for the sensor 108.

The tracking system 100 proceeds to step 6522 in response to determiningthat the sensor 108 has not been rotated. In this case, the trackingsystem 100 does not update the homography 118 based on a rotation of thesensor 108. Since the sensor 108 has not been rotated, this means thatthe homography 118 that is associated with the sensor 108 is stillvalid. The tracking system 100 proceeds to step 6518 in response todetermining that the sensor 108 has been rotated. In this case, thetracking system 100 determines to update the homography 118 based on therotation of the sensor 108. Since the sensor 108 has been rotated, thismeans that the homography 118 that is associated with the sensor 108 isno longer valid.

At step 6518, the tracking system 100 determines rotation coefficients6706 for the sensor 108 based on the rotation of the sensor 108. Therotation coefficients 6706 identify a rotational orientation of thesensor 108 with respect to the ground plane 104, for example, the x-yplane of the ground plane 104. The rotation coefficients 6706 comprisean x-axis rotation value, a y-axis rotation value, and/or a z-axisrotation value. The rotation coefficients 6706 may be in degrees orradians.

At step 6520, the tracking system 100 updates the homography 118 for thesensor 108 by applying the rotation coefficients 6706 to the homography118. The tracking system 100 may use the rotation coefficients 6706 inthe transformation matrix 6702 that can be applied to the homography118. Returning to the example in FIG. 67 , Rx corresponds with thex-axis rotation value, Ry corresponds with the y-axis rotation value,and Rz corresponds with the z-axis rotation value. The tracking system100 includes the rotation coefficients 6706 within the transformationmatrix 6702 and then uses the uses matrix multiplication to apply therotation coefficients 6706 and update the homography 118. The trackingsystem 100 may set the translation coefficients 6704 to a value of zerowhen no translation is being applied to the homography 118.

In one embodiment, the tracking system 100 may first populate thetranslation matrix 6702 with the translation coefficients 6704 and therotation coefficients 6706 and then use matrix multiplication tosimultaneously apply the translation coefficients 6704 and the rotationcoefficients 6706 to the homography 118. After updating the homography118, the tracking system 100 may store a new association between thesensor 108, the updated homography 118, the current position of thesensor 108, and the rotation angle of the sensor 108.

At step 6522, the tracking system 100 determines whether to continuemonitoring the position of the sensor 108. In one embodiment, thetracking system 100 may be configured to continuously monitor theposition of the sensor 108. In another embodiment, the tracking system100 may be configured to check the position and orientation of thesensor 108 in response to a user input from a technician. In this case,the tracking system 100 may discontinue monitoring the position of thesensor 108 until another user input is provided by a technician. Inother embodiments, the tracking system 100 may use any other suitablecriteria for determining whether to continue monitoring the position ofthe sensor 108.

The tracking system 100 returns to step 6508 in response to determiningto continue monitoring the position of the sensor. In this case, thetracking system 100 will return to step 6508 to continue monitoring forchanges in the position and orientation of the sensor 108. Otherwise,the tracking system 100 terminates method 6500 in response todetermining to discontinue monitoring the position of the sensor. Inthis case, the tracking system 100 will suspend monitoring for changesin the position and orientation of the sensor 108 and will terminatemethod 6500.

Homography Error Correction Overview

FIGS. 68-75 provide various embodiments of homography error correctiontechniques. More specifically, FIGS. 68 and 69 provide an example of ahomography error correction process based on a location of sensor 108.FIGS. 70 and 71 provide an example of a homography error correctionprocess based on distance measurements using a sensor 108. FIGS. 72 and73 provide an example of a homography error correction process based ona disparity mapping using adjacent sensors 108. FIGS. 74 and 75 providean example of a homography error correction process based on distancemeasurements using adjacent sensors 108. The tracking system 100 mayemploy one or more of these homography error correction techniques todetermine whether a homography 118 is within the accuracy tolerances ofthe system 100. When a homogaphy 118 is beyond the accuracy tolerancesof the system 100, the ability of the system 100 to accurately trackpeople and objects may decline which may reduce the overall performanceof the system 100. When the tracking system 100 determines that thehomography 118 is beyond the accuracy tolerances of the system 100, thetracking system will recompute the homography 118 to improve itsaccuracy.

Homography Error Correction Process Based on Sensor Location

FIG. 68 is a flowchart of an embodiment of a homography error correctionmethod 6800 for the tracking system 100. The tracking system 100 mayemploy method 6800 to check whether a homography 118 of a sensor 108 isproviding results within the accuracy tolerances of the system 100. Thisprocess generally involves using a homography 118 to estimate a physicallocation (i.e. an (x,y) coordinate in the global plane 104) of a sensor108. The tracking system 100 then compares the estimated physicallocation of the sensor 108 to the actual physical location of the sensor108 to determine whether the results provided using the homography 118are within the accuracy tolerances of the system 100. In the event thatthe results provided using the homography 118 is outside of the accuracytolerances of the system 100, the tracking system 100 will recompute thehomography 118 to improve its accuracy.

At step 6802, the tracking system 100 receives a frame 302 from a sensor108. Referring to FIG. 69 as an example, the sensor 108 is positionedwithin a space 102 (e.g. a store) with an overhead view of the space102. The sensor 108 is configured to capture frames 302 of the globalplane 104 for at least a portion of the space 102. In this example, amarker 304 is positioned within the field of view of the sensor 108. Inone embodiment, the marker 304 is positioned to be in the center of thefield of view of the sensor 108. In this configuration, the marker 304is aligned with a centroid or the center of the sensor 108. The trackingsystem 100 receives a frame 302 from the sensor 108 that includes themarker 304.

At step 6804, the tracking system 100 identifies a pixel location 6908within the frame 302. In one embodiment, the tracking system 100 usesobject detection to identify the marker 304 within the frame 302. Forexample, the tracking system 100 may search the frame 302 for knownfeatures (e.g. shapes, patterns, colors, text, etc.) that correspondwith the marker 304. In this example, the tracking system 100 mayidentify a shape in the frame 302 that corresponds with the marker 304.In other embodiments, the tracking system 100 may use any other suitabletechnique to identify the marker 304 within the frame 302. Afterdetecting the marker 304, the tracking system 100 identifies a pixellocation 6908 within the frame 302 that corresponds with the marker 304.In one embodiment, the pixel location 6908 corresponds with a pixel inthe center of the frame 302.

At step 6806, the tracking system 100 determines an estimated sensorlocation 6902 using a homography 118 that is associated with the sensor108. Here, the tracking system 100 uses a homography 118 that isassociated with the sensor 108 to determine an (x,y) coordinate in theglobal plane 104 for the marker 304. The homography 118 is configured totranslate between pixel locations in the frame 302 and (x,y) coordinatesin the global plane 104. The homography 118 is configured similarly tothe homography 118 described in FIGS. 2-5B. As an example, the trackingsystem 100 may identify the homography 118 that is associated with thesensor 108 and may use matrix multiplication between the homography 118and the pixel location of the marker 304 to determine an (x,y)coordinate for the marker 304 in the global plane 104. Since the marker304 is aligned with the centroid of the sensor 108, the (x,y) coordinateof the marker 304 also corresponds with the (x,y) coordinate for thesensor 108. This means that the tracking system 100 can use the (x,y)coordinate of the marker 304 as the estimated sensor location 6902.

At step 6808, the tracking system 100 determines an actual sensorlocation 6904 for the sensor 108. In one embodiment, the tracking system100 may employ position sensors 5610 that are configured to output thelocation of the sensor 108. The position sensor 5610 may be configuredsimilarly to the position sensors 5610 described in FIG. 56 . In thiscase, the position sensor 5610 may output an (x,y) coordinate for thesensor 108 that indicates where the sensor 108 is physically locatedwith respect to the global plane 104. In other embodiments, the trackingsystem 100 may be configured to receive location information (i.e. an(x,y) coordinate) for the sensor 108 from a technician or using anyother suitable technique.

At step 6810, the tracking system 100 determines a location difference6906 between the estimated sensor location 6902 and the actual sensorlocation 6904. The location difference 6906 is in real-world units andidentifies a physical distance between the estimated sensor location6902 and the actual sensor location 6904 with respect to the globalplane 104. As an example, the tracking system 100 may determine thelocation difference 6906 by determining a Euclidian distance between the(x,y) coordinate corresponding with the estimated sensor location 6902and the (x,y) coordinate corresponding with the actual sensor location6904. In other examples, the tracking system 100 may determine thelocation difference 6906 using any other suitable type of technique.

At step 6812, the tracking system 100 determines whether the locationdifference 6906 exceeds a difference threshold level 6910. Thedifference threshold level 6910 corresponds with an accuracy tolerancelevel for a homography 118. Here, the tracking system 100 compares thelocation difference 6906 to the difference threshold level 6910 todetermine whether the location difference 6906 is less than or equal tothe difference threshold level 6910. The difference threshold level 6910is in real-world units and identifies a physical distance within theglobal plane 104. For example, the difference threshold level 6910 maybe fifteen millimeters, one hundred millimeters, six inches, one foot,or any other suitable distance.

When the location difference 6906 is less than or equal to thedifference threshold level 6910, this indicates that the distancebetween the estimated sensor location 6902 and the actual sensor lotion6904 is within the accuracy tolerance for the system. In the exampleshown in FIG. 69 , the location difference 6906 is less than thedifference threshold level 6910. In this case, the tracking system 100determines that the homography 118 is within accuracy tolerances andthat the homography 118 does not need to be recomputed. The trackingsystem 100 terminates method 6800 in response to determining that thelocation difference 6906 does not exceed the difference threshold value.

When location difference 6906 exceeds the difference threshold level6910, this indicates that the distance between the estimated sensorlocation 6902 and the actual sensor location 6904 is too great toprovide accurate results using the current homography 118. In this case,the tracking system 100 determines that the homography 118 is inaccurateand that the homography 118 should be recomputed to improve accuracy.The tracking system 100 proceeds to step 6814 in response to determiningthat the location difference 6906 exceeds the difference thresholdvalue.

At step 6814, the tracking system 100 recomputes the homography 118 forthe sensor 108. The tracking system 100 may recompute the homography 118using any of the previously described techniques for generating ahomography 118. For example, the tracking system 100 may generate ahomography 118 using the process described in FIGS. 2 and 6 . Afterrecomputing the homography 118, the tracking system 100 associates thenew homography 118 with the sensor 108.

Homography Error Correction Process Based on Distance Measurements

FIG. 70 is a flowchart of another embodiment of a homography errorcorrection method 7000 for the tracking system 100. The tracking system100 may employ method 7000 to check whether a homography 118 of a sensor108 is providing results within the accuracy tolerances of the system100. This process generally involves using a homography 118 to compute adistance between two markers 304 that are within the field of view ofthe sensor 108. The tracking system 100 then compares the computeddistance to the actual distance between the markers 304 to determinewhether the results provided using the homography 118 are within theaccuracy tolerances of the system 100. In the event that the resultsprovided using the homography 118 is outside of the accuracy tolerancesof the system 100, the tracking system 100 will recompute the homography118 to improve its accuracy.

At step 7002, the tracking system 100 receives a frame 302 from a sensor108. Referring to FIG. 71 as an example, the sensor 108 is positionedwithin a space 102 (e.g. a store) with an overhead view of the space102. The sensor 108 is configured to capture frames 302 of the globalplane 104 for at least a portion of the space 102. In this example, afirst marker 304A and a second marker 304B are positioned within thefield of view of the sensor 108. The tracking system 100 receives aframe 302 from the sensor 108 that includes the first marker 304A andthe second marker 304B.

At step 7004, the tracking system 100 identifies a first pixel location7102 for a first marker 304A within the frame 302. In one embodiment,the tracking system 100 may use object detection to identify the firstmarker 304A within the frame 302. For example, the tracking system 100may search the frame 302 for known features (e.g. shapes, patterns,colors, text, etc.) that correspond with the first marker 304A. In thisexample, the tracking system 100 may identify a shape in the frame 302that corresponds with the first marker 304A. In other embodiments, thetracking system 100 may use any other suitable technique to identify thefirst marker 304A within the frame 302. After detecting the first marker304A, the tracking system 100 identifies a pixel location 7102 withinthe frame 302 that corresponds with the first marker 304A.

At step 7006, the tracking system 100 identifies a second pixel location7104 for a second marker 304B within the frame 302. The tracking system100 may use a process similar to the process described in step 7004 toidentify the second pixel location 7104 for the second marker 304Bwithin the frame 302.

At step 7008, the tracking system 100 determines a first (x,y)coordinate 7106 for the first marker 304A by applying a homography 118to the first pixel location 7102. Here, the tracking system 100 uses ahomography 118 that is associated with the sensor 108 to determine an(x,y) coordinate in the global plane 104 for the first marker 304A. Thehomography 118 is configured to translate between pixel locations in theframe 302 and (x,y) coordinates in the global plane 104. The homography118 is configured similar to the homography 118 described in FIGS. 2-5B.As an example, the tracking system 100 may identify the homography 118that is associated with the sensor 108 and may use matrix multiplicationbetween the homography 118 and the pixel location of the marker 304 todetermine an (x,y) coordinate 7102 for the first marker 304A in theglobal plane 104.

At step 7010, the tracking system 100 determines a second (x,y)coordinate 7108 for the second marker 304B by applying the homography118 to the second pixel location 7104. The tracking system 100 may use aprocess similar to the process described in step 7008 to determines asecond (x,y) coordinate 7108 for the second marker 304B.

At step 7012, the tracking system 100 determines an estimated distance7110 between the first (x,y) coordinate 7106 and the second (x,y)coordinate 7108. The estimated distance 7110 is in real-world units andidentifies a physical distance between the first marker 304A and thesecond marker 304B with respect to the global plane 104. As an example,the tracking system 100 may determine the estimated distance 7110 bydetermining a Euclidian distance between the first (x,y) coordinate 7106and the second (x,y) coordinate 7108. In other examples, the trackingsystem 100 may determine the estimated distance 7110 using any othersuitable type of technique.

At step 7014, the tracking system 100 determines an actual distance 7112between the first marker 304A and the second marker 304B. The actualdistance 7112 is in real-world units and identifies the actual physicaldistance between the first marker 304A and the second marker 304B withrespect to the global plane 104. The tracking system 100 may beconfigured to receive an actual distance 7112 between the first marker304A and the second marker 304B from a technician or using any othersuitable technique.

At step 7016, the tracking system 100 determines a distance difference7114 between the estimated distance 7110 and the actual distance 7112.The distance difference 7114 indicates a measurement difference betweenthe estimated distance 7110 and the actual distance 7112. The distancedifference 7114 is in real-world units and identifies a physicaldistance within the global plane 104. In one embodiment, the trackingsystem 100 may use the absolute value of the difference between theestimated distance 7110 and the actual distance 7112 as the distancedifference 7114.

At step 7018, the tracking system 100 determines whether the distancedifference 7114 exceeds a difference threshold value 7116. Thedifference threshold level 7116 corresponds with an accuracy tolerancelevel for a homography 118. Here, the tracking system 100 compares thedistance difference 7114 to the difference threshold level 7116 todetermine whether the distance difference 7114 is less than or equal tothe difference threshold level 7116. The difference threshold level 7116is in real-world units and identifies a physical distance within theglobal plane 104. For example, the difference threshold level 7116 maybe fifteen millimeters, one hundred millimeters, six inches, one foot,or any other suitable distance.

When the distance difference 7114 is less than or equal to thedifference threshold level 7116, this indicates the difference betweenthe estimated distance 7110 and the actual distance 7112 is within theaccuracy tolerance for the system. In the example shown in FIG. 71 , thedistance difference 7114 is less than difference threshold level 7116.In this case, the tracking system 100 determines that the homography 118is within accuracy tolerances and that the homography 118 does not needto be recomputed. The tracking system 100 terminates method 7000 inresponse to determining that the distance difference 7114 does notexceed the difference threshold value 7116.

When distance difference 7114 exceeds the difference threshold level7116, this indicates that the difference between the estimated distance7110 and the actual distance 7112 is too great to provide accurateresults using the current homography 118. In this case, the trackingsystem 100 determines that the homography 118 is inaccurate and that thehomography 118 should be recomputed to improve accuracy. The trackingsystem 100 proceeds to step 7020 in response to determining that thedistance difference 7114 exceeds the difference threshold value 7116.

At step 7020, the tracking system 100 recomputes the homography 118 forthe sensor 108. The tracking system 100 may recompute the homography 118using any of the previously described techniques for generating ahomography 118. For example, the tracking system 100 may generate ahomography 118 using the process described in FIGS. 2 and 6 . Afterrecomputing the homography 118, the tracking system 100 associates thenew homography 118 with the sensor 108.

Homography Error Correction Process Using Stereoscopic Vision Based on aDisparity Mapping

FIGS. 72-75 are embodiments of homography error correction methods usingadjacent sensors 108 in a stereoscopic sensor configuration. In astereoscopic sensor configuration, the disparity between similar pointson the frames 302 from each sensor 108 can be used to 1) correct anexisting homography 118 or 2) generate a new homography 118. In thefirst case, the tracking system 100 may correct an existing homography118 when the distance between two sensors 108 is known and the distancesbetween a series of similar points in the real-world (e.g. the globalplane 104) are also known. In this case, the distance between similarpoints can be found by calculating a disparity mapping 7308. In oneembodiment, a disparity mapping 7308 can be defined by the followingexpressions:

$D_{x} = {{\overset{\_}{P}}_{x} = {{P_{xa} - P_{xb}} = {f\frac{d_{a,b}}{G_{z}}}}}$$D_{y} = {{\overset{\_}{P}}_{y} = {{P_{ya} - P_{yb}} = {f\frac{d_{a,b}}{G_{z}}}}}$$D = \sqrt{D_{x}^{2} + D_{y}^{2}}$where D is the disparity mapping 7308, P is the location of a real-worldpoint, Px is the distance between a similar point in two cameras (e.g.camera ‘a’ and camera ‘b’), P_(xa) is the x-coordinate of a point withrespect to camera ‘a,’ P_(xb) is the x-coordinate of a similar pointwith respect to camera ‘b,’ P_(ya) is the y-coordinate of a point withrespect to camera ‘a,’ P_(yb) is the y-coordinate of a similar pointwith respect to camera ‘b,’ f is the focal length, d_(a,b) is the realdistance between a camera ‘a’ and camera ‘b,’ G_(z) is the verticaldistance between a camera to a real-world point. The disparity mapping7308 may be used to determine 3D world points for a marker 304 using thefollowing expression:

$G_{x,y} = \frac{d_{a,b}P_{x,y}^{a}}{D}$where G_(x,y) is the global position of a marker 304. Using thisprocess, the tracking system 100 may compare the disparity betweenhomography projected distances between points and stereo estimateddistances between the points to determine the accuracy of thehomographies 118 for the adjacent sensors 108.

In the second case, the tracking system 100 may generate a newhomography 118 when the distance between two sensors 108 is known. Inthis case, the real distances between similar points are also known. Thetracking system 100 may use the stereoscopic sensor configuration tocalculate a homography 118 for each sensor 108 to the global plane 104.Since G_(x,y) is known for each sensor 108, the tracking system 100 mayuse this information to compute a homography 118 between the sensors 108and the global plane 104. For example, the tracking system 100 may usethe stereoscopic sensor configuration to determine 3D point locationsfor a set of markers 104. The tracking system 100 may then determine theposition of the markers 304 with respect to the global plane 104 usingG_(x,y). The tracking system 100 may then use the 3D point locations fora set of markers 104 and the position of the markers 304 with respect tothe global plane 104 to compute a homography 118 for a sensor 108.

FIG. 72 is a flowchart of another embodiment of a homography errorcorrection method 7200 for the tracking system 100. The tracking system100 may employ method 7200 to check whether the homographies 118 of apair of sensors 108 are providing results within the accuracy tolerancesof the system 100. This process generally involves using homographies118 to determine first pixel location within a frame 302 for a marker304 that is within the field of view of a pair of adjacent sensors 108.The tracking system 100 then determines a second pixel location withinthe frame 302 using a disparity mapping 7308. The disparity mapping 7308is configured to map between pixel locations 7310 in frames 302A fromthe first sensor 108 and pixel locations 7312 in frames 302B from thesecond sensor 108. The tracking system 100 then computes a distancebetween the first pixel location and the second pixel location todetermine whether the results provided using the homographies 118 arewithin the tolerances of the system 100. In the event that the resultsprovided using the homographies 118 are outside of the accuracytolerances of the system 100, the tracking system 100 will recompute thehomographies 118 to improve their accuracy.

At step 7202, the tracking system 100 receives a frame 302A from a firstsensor 108. In one embodiment, the first sensor 108 is positioned withina space 102 (e.g. a store) with an overhead view of the space 102. Thesensor 108 is configured to capture frames 302A of the global plane 104for at least portion of the space 102. In this example, a marker 304 ispositioned within the field of view of the first sensor 108. Referringto FIG. 73 as an example, the tracking system 100 receives a frame 302Afrom the first sensor 108 that includes the marker 304.

At step 7204, the tracking system 100 identifies a first pixel location7302 for the marker 304 within the frame 302A. In one embodiment, thetracking system 100 may use object detection to identify the marker 304within the frame 302A. For example, the tracking system 100 may searchthe frame 302A for known features (e.g. shapes, patterns, colors, text,etc.) that correspond with the marker 304. In this example, the trackingsystem 100 may identify a shape in the frame 302A that corresponds withthe marker 304. In other embodiments, the tracking system 100 may useany other suitable technique to identify the marker 304 within the frame302A. After detecting the marker 304, the tracking system 100 identifiesa pixel location 7302 within the frame 302A that corresponds with themarker 304.

At step 7206, the tracking system 100 determines an (x,y) coordinate byapplying a first homography 118 to the first pixel location 7302. Here,the tracking system 100 uses a first homography 118 that is associatedwith the first sensor 108 to determine an (x,y) coordinate in the globalplane 104 for the marker 304. The first homography 118 is configured totranslate between pixel locations in the frame 302A and (x,y)coordinates in the global plane 104. The first homography 118 isconfigured similar to the homography 118 described in FIGS. 2-5B. As anexample, the tracking system 100 may identify the first homography 118that is associated with the sensor 108 and may use matrix multiplicationbetween the first homography 118 and the pixel location 7302 of themarker 304 to determine an (x,y) coordinate for the marker 304 in theglobal plane 104.

At step 7208, the tracking system 100 identifies a second pixel location7304 by applying a second homography 118 to the (x,y) coordinate. Thesecond pixel location 7304 is a pixel location within a frame 302B of asecond sensor 108. For example, a second sensor 108 may be positionedadjacent to the first sensor 108 such that frames 302A from the firstsensor 108 at least partially overlap with frames 302B from the secondsensor 108. The tracking system 100 uses a second homography 118 that isassociated with the second sensor 108 to determine a pixel location 7304based on the determined (x,y) coordinate of the marker 304. The secondhomography 118 is configured to translate between pixel locations in theframe 302B and (x,y) coordinates in the global plane 104. The secondhomography 118 is configured similarly to the homography 118 describedin FIGS. 2-5B. As an example, the tracking system 100 may identify thesecond homography 118 that is associated with the second sensor 108B andmay use matrix multiplication between the second homography 118 and the(x,y) coordinate of the marker 304 to determine the second pixellocation 7304 within the second frame 302B.

At step 7210, the tracking system 100 identifies a third pixel location7306 by applying a disparity mapping 7308 to the first pixel location7302. In FIG. 73 , the disparity mapping 7308 is shown as a table. Inother examples, the disparity mapping 7308 may be a mapping functionthat is configured to translate between pixel locations 7310 in frames302A from the first sensor 108 and pixel locations 7312 in frames 302Bfrom the second sensor 108. The tracking system 100 uses the first pixellocation 7302 as input for the disparity mapping 7308 to determine athird pixel location 7306 within the second frame 302B.

At step 7212, the tracking system 100 determines a distance difference7314 between the second pixel location 7304 and the third pixel location7306. The distance difference 7314 in in pixel units and identifies thepixel distance between the second pixel location 7304 and the thirdpixel location 7306. As an example, the tracking system 100 maydetermine the distance difference 7314 by determining a Euclideandistance between the second pixel location 7304 and the third pixellocation 7306. In other examples, the tracking system 100 may determinethe distance difference 7314 using any other suitable type of technique.

At step 7214, the tracking system 100 determines whether the distancedifference 7314 exceeds a difference threshold value 7316. Thedifference threshold level 7316 corresponds with an accuracy tolerancelevel for a homography 118. Here, the tracking system 100 compares thedistance difference 7314 to the difference threshold level 7316 todetermine whether the distance difference 7314 is less than or equal tothe difference threshold level 7316. The difference threshold level 7316is in pixel units and identifies a distance within a frame 302. Forexample, the difference threshold level 7316 may be one pixel, fivepixel, ten pixels, or any other suitable distance.

When the distance difference 7314 is less than or equal to thedifference threshold level 7316, this indicates the difference betweenthe second pixel location 7304 and the third pixel location 7306 iswithin the accuracy tolerance for the system. In the example shown inFIG. 73 , the distance difference 7314 is less than difference thresholdlevel 7316. In this case, the tracking system 100 determines that thehomographies 118 for the first sensor 108 and the second sensor 108 arewithin accuracy tolerances and that the homographies 118 do not need tobe recomputed. The tracking system 100 terminates method 7200 inresponse to determining that the distance difference 7314 does notexceed the difference threshold value 7316.

When distance difference 7314 exceeds the difference threshold level7316, this indicates that the difference between the second pixellocation 7304 and the third pixel location 7306 is too great to provideaccurate results using the current homographies 118. In this case, thetracking system 100 determines that at least one of the homographies 118for the first sensor 108 or the second sensor 108 is inaccurate and thatthe homographies 118 should be recomputed to improve accuracy. Thetracking system 100 proceeds to step 7216 in response to determiningthat the distance difference 7314 exceeds the difference threshold value7316.

At step 7216, the tracking system 100 recomputes the homography 118 forthe first sensor 108 and/or the second sensor 108. The tracking system100 may recompute the homography 118 for the first sensor 108 and/or thesecond sensor 108 using any of the previously described techniques forgenerating a homography 118. For example, the tracking system 100 maygenerate a homography 118 using the process described in FIGS. 2 and 6 .After recomputing the homography 118, the tracking system 100 associatesthe new homography 118 with the corresponding sensor 108.

Homography Error Correction Process Using Stereoscopic Vision Based onDistance Measurements

FIG. 74 is a flowchart of another embodiment of a homography errorcorrection method 7400 for the tracking system 100. The tracking system100 may employ method 7400 to check whether the homographies 118 of apair of sensors 108 are providing results within the accuracy tolerancesof the system 100. This process generally involves using homographies118 to compute a distance between two markers 304 using adjacent sensors108. The tracking system 100 then compares the computed distance to theactual distance between the markers 304 to determine whether the resultsprovided using the homographies 118 are within the accuracy tolerancesof the system 100. In the event that the results provided using thehomographies 118 are outside of the accuracy tolerances of the system100, the tracking system 100 will recompute the homographies 118 toimprove their accuracy.

At step 7402, the tracking system 100 receives a first frame 302A from afirst sensor 108A. Referring to FIG. 75 as an example, the first sensor108A is positioned within a space 102 (e.g. a store) with an overheadview of the space 102. The first sensor 108A is configured to captureframes 302A of the global plane 104 for at least a portion of the space102. In this example, a first marker 304A and a second marker 304B arepositioned within the field of view of the first sensor 108A. In oneembodiment, the first marker 304A may be positioned in the center of thefield of view of the first sensor 108A. The tracking system 100 receivesa frame 302A from the first sensor 108A that includes the first marker304A and the second marker 304B.

At step 7404, the tracking system 100 identifies a first pixel location7502 for the first marker 304A within the first frame 302A. In oneembodiment, the tracking system 100 may use object detection to identifythe first marker 304A within the first frame 302A. For example, thetracking system 100 may search the first frame 302A for known features(e.g. shapes, patterns, colors, text, etc.) that correspond with thefirst marker 304A. In this example, the tracking system 100 may identifya shape in the first frame 302A that corresponds with the first marker304A. In other embodiments, the tracking system 100 may use any othersuitable technique to identify the first marker 304A within the firstframe 302A. After detecting the first marker 304A, the tracking system100 identifies a pixel location 7502 within the first frame 302A thatcorresponds with the first marker 304A. In one embodiment, the pixellocation 7502 may correspond with a pixel in the center of the firstframe 302A.

At step 7406, the tracking system 100 determines a first (x,y)coordinate 7504 for the first marker 304A by applying a first homography118 to the first pixel location 7502. The tracking system 100 uses afirst homography 118 that is associated with the first sensor 108A todetermine a first (x,y) coordinate 7504 in the global plane 104 for thefirst marker 304A. The first homography 118 is configured to translatebetween pixel locations in the frame 302A and (x,y) coordinates in theglobal plane 104. The first homography 118 is configured similar to thehomography 118 described in FIGS. 2-5B. As an example, the trackingsystem 100 may identify the first homography 118 that is associated withthe first sensor 108A and may use matrix multiplication between thefirst homography 118 and the first pixel location 7502 to determine thefirst (x,y) coordinate 7504 for the first marker 304A in the globalplane 104. In the example, where the pixel location 7502 correspondswith a pixel in the center of the first frame 302A, the first (x,y)coordinate 7504 may correspond with an estimated location for the firstsensor 108A.

At step 7408, the tracking system 100 receives a second frame 302B froma second sensor 108B. Returning to the example in FIG. 75 , the secondsensor 108B is also positioned within the space 102 with an overheadview of the space 102. The second sensor 108B is configured to captureframes 302B of the global plane 104 for at least portion of the space102. The second sensor 108B is positioned adjacent to the first sensor108A such that frames 302A from the first sensor 108A at least partiallyoverlap with frames 302B from the second sensor 108B. The first marker304A and the second marker 304B are positioned within the field of viewof the second sensor 108B. In one embodiment, the second marker 304B maybe positioned in the center of the field of view of the second sensor108B. The tracking system 100 receives a frame 302B from the secondsensor 108B that includes the first marker 304A and the second marker304B.

At step 7410, the tracking system 100 identifies a second pixel location7506 for the second marker 304B within the second frame 302B. Thetracking system 100 identifies the second pixel location 7506 for thesecond marker 304B using a process similar to the process described instep 7404. In one embodiment, the pixel location 7506 may correspondwith a pixel in the center of the second frame 302B.

At step 7412, the tracking system 100 determines a second (x,y)coordinate 7508 for the second marker 304B by applying a secondhomography 118 to the second pixel location 7506. The tracking system100 uses a second homography 118 that is associated with the secondsensor 108B to determine a second (x,y) coordinate 7508 in the globalplane 104 for the second marker 304B. The second homography 118 isconfigured to translate between pixel locations in the frame 302B and(x,y) coordinates in the global plane 104. The second homography 118 isconfigured similarly to the homography 118 described in FIGS. 2-5B. Asan example, the tracking system 100 may identify the second homography118 that is associated with the second sensor 108B and may use matrixmultiplication between the second homography 118 and the second pixellocation 7506 to determine the second (x,y) coordinate 7508 for thesecond marker 304B in the global plane 104. In the example, where thepixel location 7506 corresponds with a pixel in the center of the secondframe 302AB, the second (x,y) coordinate 7508 may correspond with anestimated location for the second sensor 108B.

At step 7414, the tracking system 100 determines a computed distance7512 between the first (x,y) coordinate 7504 and the second (x,y)coordinate 7508. The computed distance 7512 is in real-world units andidentifies a physical distance between the first marker 304A and thesecond marker 304B with respect to the global plane 104. As an example,the tracking system 100 may determine the computed distance 7512 bydetermining a Euclidian distance between the first (x,y) coordinate 7504and the second (x,y) coordinate 7508. In other examples, the trackingsystem 100 may determine the computed distance 7512 using any othersuitable type of technique.

At step 7416, the tracking system 100 determines an actual distance 7514between the first marker 304A and the second marker 304B. The actualdistance 7514 is in real-world units and identifies the actual physicaldistance between the first marker 304A and the second marker 304B withrespect to the global plane 104. The tracking system 100 may beconfigured to receive an actual distance 7514 between the first marker304A and the second marker 304B from a technician or using any othersuitable technique.

At step 7418, the tracking system 100 determines a distance difference7516 between the computed distance 7512 and the actual distance 7514.The distance difference 7516 indicates a measurement difference betweenthe computed distance 7512 and the actual distance 7514. The distancedifference 7516 is in real-world units and identifies a physicaldistance within the global plane 104. In one embodiment, the trackingsystem 100 may use the absolute value of the difference between thecomputed distance 7512 and the actual distance 7514 as the distancedifference 7516.

At step 7420, the tracking system 100 determines whether the distancedifference 7516 exceeds a difference threshold level 7518. Thedifference threshold level 7516 corresponds with an accuracy tolerancelevel for a homography 118. Here, the tracking system 100 compares thedistance difference 7516 to the difference threshold level 7518 todetermine whether the distance difference 7516 is less than or equal tothe difference threshold level 7518. The difference threshold level 7518is in real-world units and identifies a physical distance within theglobal plane 104. For example, the difference threshold level 7518 maybe fifteen millimeters, one hundred millimeters, six inches, one foot,or any other suitable distance.

When the distance difference 7516 is less than or equal to thedifference threshold level 7518, this indicates the difference betweenthe computed distance 7512 and the actual distance 7514 is within theaccuracy tolerance for the system. In the example shown in FIG. 75 , thedistance difference 7516 is less than difference threshold level 7518.In this case, the tracking system 100 determines that the homographies118 for the first sensor 108A and the second sensor 108B are withinaccuracy tolerances and that the homographies 118 do not need to berecomputed. The tracking system 100 terminates method 7400 in responseto determining that the distance difference 7516 does not exceed thedifference threshold level 7518.

When distance difference 7516 exceeds the difference threshold level7518, this indicates that the difference between the computed distance7512 and the actual distance 7514 is too great to provide accurateresults using the current homographies 118. In this case, the trackingsystem 100 determines that at least one of the homographies 118 for thefirst sensor 108A or the second sensor 108B is inaccurate and that thehomographies 118 should be recomputed to improve accuracy. The trackingsystem 100 proceeds to step 7422 in response to determining that thedistance difference 7516 exceeds the difference threshold level 7518.

At step 7422, the tracking system 100 recomputes the homography 118 forthe first sensor 108A and/or the second sensor 108B. The tracking system100 may recompute the homography 118 for the first sensor 108A and/orthe second sensor 108B using any of the previously described techniquesfor generating a homography 118. For example, the tracking system 100may generate a homography 118 using the process described in FIGS. 2 and6 . After recomputing the homography 118, the tracking system 100associates the new homography 118 with the corresponding sensor 108.

While the preceding examples and explanations are described with respectto particular use cases within a retail environment, one of ordinaryskill in the art would readily appreciate that the previously describedconfigurations and techniques may also be applied to other applicationsand environments. Examples of other applications and environmentsinclude, but are not limited to, security applications, surveillanceapplications, object tracking applications, people trackingapplications, occupancy detection applications, logistics applications,warehouse management applications, operations research applications,product loading applications, retail applications, roboticsapplications, computer vision applications, manufacturing applications,safety applications, quality control applications, food distributingapplications, retail product tracking applications, mappingapplications, simultaneous localization and mapping (SLAM) applications,3D scanning applications, autonomous vehicle applications, virtualreality applications, augmented reality applications, or any othersuitable type of application.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

To aid the Patent Office, and any readers of any patent issued on thisapplication in interpreting the claims appended hereto, applicants notethat they do not intend any of the appended claims to invoke 35 U.S.C. §112(f) as it exists on the date of filing hereof unless the words “meansfor” or “step for” are explicitly used in the particular claim.

The invention claimed is:
 1. A system, comprising: a beverage machineoperable to dispense a beverage; an image sensor positioned such that afield-of-view of the image sensor encompasses at least a portion of thebeverage machine, wherein the image sensor is configured to generateangled-view images of at least a portion of the beverage machine,wherein the portion includes a first zone associated with operating thebeverage machine to dispense the beverage and a second zone in which acup is placed to receive the beverage from the beverage machine; and abeverage assignment subsystem coupled to the image sensor, the beverageassignment subsystem comprising at least one processor configured to:receive an image feed comprising frames of the angled-view imagesgenerated by the image sensor; detect an event associated with an objectentering one or both of the first zone and the second zone; determine,in a first one or more image frames associated with a start of thedetected event, that both a hand of a person enters the first zone andthe cup is placed in the second zone; detect that the cup is removedfrom the second zone; following detecting that the cup is removed fromthe second zone, determine a length of time during which the cupremained in the second zone; if the cup remained in the second zone forat least a threshold time, assign the beverage to the person whose handentered the first zone; if the cup did not remain in in the second zonefor at least the threshold time, do not assign the beverage to theperson whose hand entered the first zone; wherein the processor isfurther configured to, in response to detecting the event, identifyframes of images from the image feed associated with the detected event,wherein the identified image frames include: the first one or more imageframes associated with the start of the detected event, wherein thefirst one or more image frames include images from the image feed from afirst predefined time period before the detected event until a secondpredefined time period after the detected event; and a second one ormore image frames associated with removal of the cup from the secondzone, wherein the second one or more image frames include images fromthe image feed from a third predefined time period before the cup isremoved from the second zone until a fourth predefined time period afterthe cup is removed from the second zone.
 2. The system of claim 1,wherein the processor is further configured to: determine a cup size ofthe cup detected in the second zone; determine a beverage type of thebeverage dispensed from the beverage machine; and assign the beverage tothe person by adding a beverage item to a digital shopping cartassociated with the person, wherein the beverage item includes anindication of the cup size and the beverage type.
 3. The system of claim1, wherein the processor is further configured to, prior to assigningthe beverage to the person if the cup remained in the second zone for atleast the threshold time, determine, based on properties of the personin the first one or more image frames associated with the start of thedetected event, a first identifier of the person whose hand entered thefirst zone.
 4. The system of claim 3, wherein the processor is furtherconfigured to, prior to assigning the beverage to the person if the cupremained in the second zone for at least the threshold time: determine,based on properties of the person in a second one or more image framesassociated with removal of the cup from the second zone, a secondidentifier of the person whose hand removed the cup from the secondzone; and determine that the second identifier is the same as the firstidentifier.
 5. The system of claim 1, wherein the processor is furtherconfigured to detect the event by determining that a portion of theperson enters one or both of the first zone and the second zone.
 6. Thesystem of claim 1, wherein: the beverage machine comprises at least onevisible marker positioned and configured to identify a position of oneor both of the first zone and the second zone; and the processor isfurther configured to: detect the at least one marker in an angled-viewimage received from the image sensor; and automatically determine, basedon the detected marker, an extent of one or both of the first zone andthe second zone.
 7. A method, comprising: receiving an image feedcomprising frames of angled-view images generated by an image sensor,wherein the image sensor is positioned such that a field-of-view of theimage sensor encompasses at least a portion of a beverage machineoperable to dispense a beverage, wherein the image sensor is configuredto generate the angled-view images of at least a portion of the beveragemachine, wherein the portion includes a first zone associated withoperating the beverage machine to dispense the beverage and a secondzone in which a cup is placed to receive the beverage from the beveragemachine; detecting an event associated with an object entering one orboth of the first zone and the second zone; determining, in a first oneor more image frames associated with a start of the detected event, thatboth a hand of a person enters the first zone and the cup is placed inthe second zone; detecting that the cup is removed from the second zone;following detecting that the cup is removed from the second zone,determining a length of time during which the cup remained in the secondzone; if the cup remained in the second zone for at least a thresholdtime, assigning the beverage to the person whose hand entered the firstzone; if the cup did not remain in in the second zone for at least thethreshold time, not assigning the beverage to the person whose handentered the first zone; in response to detecting the event, identifyingframes of images from the image feed associated with the detected event,wherein the identified image frames include: the first one or more imageframes associated with the start of the detected event, wherein thefirst one or more image frames include images from the image feed from afirst predefined time period before the detected event until a secondpredefined time period after the detected event; and a second one ormore image frames associated with removal of the cup from the secondzone, wherein the second one or more image frames include images fromthe image feed from a third predefined time period before the cup isremoved from the second zone until a fourth predefined time period afterthe cup is removed from the second zone.
 8. The method of claim 7,further comprising: determining a cup size of the cup detected in thesecond zone; determining a beverage type of the beverage dispensed fromthe beverage machine; and assigning the beverage to the person by addinga beverage item to a digital shopping cart associated with the person,wherein the beverage item includes an indication of the cup size and thebeverage type.
 9. The method of claim 7, further comprising, prior toassigning the beverage to the person if the cup remained in the secondzone for at least the threshold time, determining, based on propertiesof the person in the first one or more image frames associated with thestart of the detected event, a first identifier of the person whose handentered the first zone.
 10. The method of claim 9, further comprising,prior to assigning the beverage to the person if the cup remained in thesecond zone for at least the threshold time: determining, based onproperties of the person in a second one or more image frames associatedwith removal of the cup from the second zone, a second identifier of theperson whose hand removed the cup from the second zone; and determiningthat the second identifier is the same as the first identifier.
 11. Themethod of claim 7, further comprising detecting the event by determiningthat a portion of the person enters one or both of the first zone andthe second zone.
 12. The method of claim 7, wherein: the beveragemachine comprises at least one visible marker positioned and configuredto identify a position of one or both of the first zone and the secondzone; and the method further comprises: detecting the at least onemarker in an angled-view image received from the image sensor; andautomatically determining, based on the detected marker, an extent ofone or both of the first zone and the second zone.
 13. A beverageassignment subsystem comprising at least one processor configured to:receive an image feed comprising frames of angled-view images generatedby an image sensor, wherein the image sensor is positioned such that afield-of-view of the image sensor encompasses at least a portion of abeverage machine operable to dispense a beverage, wherein the imagesensor is configured to generate the angled-view images of at least aportion of the beverage machine, wherein the portion includes a firstzone associated with operating the beverage machine to dispense thebeverage and a second zone in which a cup is placed to receive thebeverage from the beverage machine; detect an event associated with anobject entering one or both of the first zone and the second zone;determine, in a first one or more image frames associated with a startof the detected event, that both a hand of a person enters the firstzone and the cup is placed in the second zone; detect that the cup isremoved from the second zone; following detecting that the cup isremoved from the second zone, determine a length of time during whichthe cup remained in the second zone; if the cup remained in the secondzone for at least a threshold time, assign the beverage to the personwhose hand entered the first zone; if the cup did not remain in in thesecond zone for at least the threshold time, do not assign the beverageto the person whose hand entered the first zone; wherein the processoris further configured to, in response to detecting the event, identifyframes of images from the image feed associated with the detected event,wherein the identified image frames include: the first one or more imageframes associated with the start of the detected event, wherein thefirst one or more image frames include images from the image feed from afirst predefined time period before the detected event until a secondpredefined time period after the detected event; and a second one ormore image frames associated with removal of the cup from the secondzone, wherein the second one or more image frames include images fromthe image feed from a third predefined time period before the cup isremoved from the second zone until a fourth predefined time period afterthe cup is removed from the second zone.
 14. The beverage assignmentsubsystem of claim 13, wherein the processor is further configured to:determine a cup size of the cup detected in the second zone; determine abeverage type of the beverage dispensed from the beverage machine; andassign the beverage to the person by adding a beverage item to a digitalshopping cart associated with the person, wherein the beverage itemincludes an indication of the cup size and the beverage type.
 15. Thebeverage assignment subsystem of claim 13, wherein the processor isfurther configured to, prior to assigning the beverage to the person ifthe cup remained in the second zone for at least the threshold time,determine, based on properties of the person in the first one or moreimage frames associated with the start of the detected event, a firstidentifier of the person whose hand entered the first zone.
 16. Thebeverage assignment subsystem of claim 15, wherein the processor isfurther configured to, prior to assigning the beverage to the person ifthe cup remained in the second zone for at least the threshold time:determine, based on properties of the person in a second one or moreimage frames associated with removal of the cup from the second zone, asecond identifier of the person whose hand removed the cup from thesecond zone; and determine that the second identifier is the same as thefirst identifier.
 17. The beverage assignment subsystem of claim 13,wherein the processor is further configured to detect the event bydetermining that a portion of the person enters one or both of the firstzone and the second zone.
 18. The beverage assignment subsystem of claim13, wherein: the beverage machine comprises at least one visible markerpositioned and configured to identify a position of one or both of thefirst zone and the second zone; and the processor is further configuredto: detect the at least one marker in an angled-view image received fromthe image sensor; and automatically determine, based on the detectedmarker, an extent of one or both of the first zone and the second zone.